Diagnosing Bimanual Robot Failures Stage by Stage

A final pass/fail result shows whether a robot completed a task, but it does not identify the behavior that produced that outcome. By attaching diagnostic measures to skill-specific stages, evaluators can locate the phase in which a bimanual policy loses alignment, slips during a grasp, coordinates its arms inconsistently or relies on them asymmetrically.

RoboEval applies this approach in simulation. Its first iteration organizes 8 base tasks and 28 variations into stages designed to challenge spatial, physical and coordination abilities. The framework evaluates outcome, efficiency, bimanual coordination and safety or stability, with fine-grained measures including trajectory smoothness and collisions with the environment.

The benchmark’s authors report that policies with similar success rates can execute the same task differently: one may struggle with alignment, while another may fail to maintain temporally consistent control across both arms. They also report that behavioral metrics correlate with success in over half of the task-metric pairs and remain informative when binary success saturates.

These results come from RoboEval’s authors and describe a simulation benchmark, not independent confirmation or a physical-robot study. Within that setting, stage-level measurements make a failure actionable by distinguishing its timing and type rather than collapsing the entire rollout into one success score.