A Shared Spatial Vocabulary Lets One Robot Model Act and Forecast

In the authors’ Spatial Language Modeling approach, one autoregressive Transformer can generate actions and predict action-conditioned future states because scene contours, goals, action targets and future states are all encoded as ordered spatial-token sequences. A shared vocabulary supplies discrete coordinate and semantic tokens, while a task-specific grammar organizes them. Under a common next-token objective, producing action-target tokens serves control, and producing future-state tokens conditioned on an action serves forecasting.

Training starts from scratch with random-play transitions, then continues with joint action and state training on expert demonstrations. During pretraining, recorded action coordinates condition the subsequent state predictions but are excluded from the prediction loss. At control time, the Transformer decodes only executable action targets and adds newly observed states to its history.

When given action trajectories, the same model also predicts successive scene states that, according to the authors, capture the geometric effects of pushing. These states are model-generated approximations conditioned on supplied actions, distinct from the new observations used to update the model during control.

The authors evaluated the method on Push-T in simulation and on a real robot. They report competitive simulation performance, plus higher task success and target coverage than the real-robot policy baselines they evaluated. Their ablations also associate joint action-state sequences with better control and random-play pretraining with further gains.

Those results remain claims from the preprint’s authors rather than independent confirmation. The available abstract provides no numerical scores or fuller experimental protocol, so the evidence supports the shared-token mechanism but does not permit a detailed assessment of the reported performance advantage.