Simulation to reality
What PhysCanvas Tests Reveal About World-Model Policy Evaluation
Across four real-world manipulation tasks, the authors report increased mean task-wise correlation between world-model and real-world success rates, followed by a higher mean real-world success rate after reward-guided policy post-training.
A realistic-looking imagined rollout is not necessarily an accurate test of a robot policy. In contact-rich manipulation, a generated scene can appear plausible while misrepresenting contact dynamics, so PhysCanvas models more than RGB video: it brings multi-view RGB, tactile images, marker displacement, task state and actions into a shared visual representation.
The model uses a video VAE and a pretrained generative backbone across those modalities. Motion images derived through forward kinematics and camera projection align action conditioning with the scene, while an initial RGB anchor and updated observations guide sequential generation. The frozen Scene, Modality and State Reward Models then supervise scene preservation, compatibility between visual and tactile predictions, and agreement with real task outcomes.
Across four real-world manipulation tasks, the authors report that State Reward Model supervision raised the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. The result therefore supports improved agreement between imagined and physical evaluations within these experiments, rather than proving that generated rollouts reproduce every aspect of contact.
The authors also used imagined interactions and rewards for GRPO policy updates. They report that post-training the π0.5 policy in the reward-guided world model increased its mean real-world success rate from a 59.1% SFT baseline to 70.8%. Within this four-task study, PhysCanvas provides evidence that a visuo-tactile world model can become a more credible proxy for physical policy evaluation when it is explicitly trained to match real task outcomes, but it does not establish that the method generalizes beyond the reported setup. Both results come from the authors’ preprint, with no independent validation in the supplied evidence.