Embodied AI models
How PRISM Multiplies Humanoid Training Examples With Counterfactual Video
PRISM generates varied videos and uses a contact-anchored real-to-sim pipeline to reconstruct human and object motions and retarget imperfect video data into physically plausible trajectories; the preprint authors also report deploying the resulting policy on a physical robot.
The authors present carrying diverse objects as an example of a humanoid loco-manipulation skill. PRISM expands a handful of real human-object interaction videos into hundreds of counterfactual clips, reconstructs their human and object motion as physically plausible trajectories, and retargets those trajectories for humanoid policy training.
The proposed framework uses video-to-video generation to introduce variation within object categories. This amplification targets a practical bottleneck identified by the authors: obtaining recordings that clearly show a person's full body and unobstructed interaction with an object.
The preprint authors report that the resulting variability allowed them to train one policy that generalized to unseen objects within each category. They say a real humanoid, using only onboard depth observations and no real-world fine-tuning, picked up, carried, and dropped novel boxes, barrels, bins, and balls across different sizes and initial configurations. The central contribution is the combination of synthetic data expansion and motion retargeting, not counterfactual video generation alone.