Data and benchmarks
How VOMMI Turns Portable RGB Recordings Into Robot Training Signals
The proposed VOMMI pipeline aligns body and hand video, corrects trajectory drift offline, and supplies local motion cues to a vision-language-action model, with physical results reported by the preprint’s authors.
Portable RGB demonstrations can train a mobile manipulation robot by synchronizing body and hand views, reconstructing their trajectories with anchored visual odometry, and converting local motion estimates into tokens that condition a vision-language-action policy during post-training.
VOMMI synchronizes body and hand views to capture navigation context and local interactions with objects.
The framework first processes each demonstration offline. Its R2-VO component uses sparse geometric anchors to refine visual-odometry trajectories, which estimate motion from images but can accumulate drift. It then represents the reconstructed movement as causal local-motion tokens spanning multiple prediction horizons. During online operation, an action-group residual adapter injects those tokens only into the vision-language-action model’s base branch.
For each task, the authors used 500 portable trajectories, held out 75 trajectories for RGB-VO evaluation, and used 200 robot demonstrations as references. They report that offline reconstruction lowered absolute trajectory error across the body and hand streams by an average of 24.6% relative to the best evaluated baseline for each stream.
The policy was post-trained only on portable demonstrations. Its authors report 18.2% lower base-velocity error than a policy trained with robot-collected demonstrations, while end-effector translation accuracy remained comparable. Across three real-robot tasks, they also report an 8.3 percentage-point improvement in mean success rate over OpenPI 0.5.
The available abstract reports a mean result across three real-robot tasks but does not provide task-by-task results. Within that limit, the preprint describes a route from portable, robot-free RGB demonstrations to motion-conditioned mobile-manipulation training.