Data and benchmarks
A Controlled Test for Spatial Grounding in Real Robots
Spa-Bench controls familiarity with objects and actions while varying spatial arrangements, then uses four diagnostic conditions to narrow down why a robot failed.
A robot that responds incorrectly to a spatial instruction has not necessarily misunderstood the language. Object and motor unfamiliarity may instead offer alternative explanations for failure, so the evaluation retains training support for relevant objects, concepts, and actions when spatial configurations change.
Change the layout, not the underlying skill
A real-robot benchmark can help distinguish spatial grounding failures from manipulation failures by changing spatial configurations while retaining training support for the relevant objects, concepts, and actions, then checking physical execution, sensitivity to wording, object-task associations, and recognition that no action is required.
Spa-Bench applies this approach by comparing in-distribution (ID) and out-of-distribution (OOD) spatial configurations. Keeping the relevant objects, concepts, and actions supported during training reduces object and motor unfamiliarity as an alternative explanation when the spatial arrangement changes.
The authors test six types of tabletop reasoning: Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Together, these tasks examine whether a policy can connect different kinds of spatial descriptions to behavior rather than merely succeed at an aggregate manipulation benchmark.
Check competing explanations directly
Four additional conditions probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. These checks matter because a robot may fail the spatially phrased command even when it can carry out the underlying manipulation, or it may act when the correct response is to do nothing.
Across the three fully evaluated policies, the authors report that matched direct instructions raised success by 28.3-40.0 percentage points. They interpret this improvement as evidence that the policies could often execute the required manipulation but struggled to ground the spatial description.
The authors report releasing 4,092 episodes collected on an SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five VLA policies. For the three fully evaluated policies, reported success ranged from 67.5-73.3% on ID configurations and from 40.7-52.0% on OOD configurations.
This design narrows the diagnosis rather than proving that every remaining error is purely linguistic. The evidence comes from one preprint, and its available abstract does not provide the detailed construction or per-task outcomes of every diagnostic condition. Within those limits, holding object and action familiarity relatively stable, varying spatial structure, and testing simpler competing explanations offers a practical way to distinguish weak spatial grounding from a robot's inability to manipulate.