Separating a Robot Task’s Effect From Its Actor

A demonstration mixes the intended change in the scene with the movements of the particular arm that produced it. The preprint proposes separating the two: the demonstrator is removed, while the task is represented by the 3D keypoint trajectories of moved objects, two points showing where each object was held and the scene’s terminal configuration. A robot can turn a demonstration into an editable program by retaining the moved objects’ 3D trajectories, their contact points and the desired final scene, then generating a new execution from the live scene.

PEWAM is a 71.5M-parameter world-action model that generates four streams: the effect, robot execution, action and terminal state. Each stream has its own flow-matching time. At inference, the requested effect program is held fixed while the model samples an execution, allowing it to re-solve the task in closed loop from the current scene rather than simply replaying the demonstrator’s motions.

Because the program consists of coordinates, it can also be edited directly. In examples reported by the authors, shifting the terminal state changes where the object is placed, while rotating the contact points changes the grasp orientation. The authors further report that one program ran on four robot arms without retraining. After an object was pushed, re-solving completed 23 of 60 episodes, compared with 6 for demonstration replay.

On held-out LIBERO-Goal tasks, a program compiled from one demonstration completed 40 of 90 episodes; the same backbone conditioned on a goal image or language, and published demonstration-conditioned methods, completed at most 19. The model exceeded the best published results on three held-out Meta-World classes, though not on the five-class mean. After fine-tuning on real demonstrations of other tasks, a Franka arm using programs compiled from single human videos completed 36 of 40 trials, versus 22 when the same backbone received the video’s last frame as a goal image.