Data and benchmarks
What a Real-Robot Test Must Control from Reset to Result
Coordinating the evaluation chain, automating resets, and classifying outcomes can make physical robot trials repeatable and open to inspection.
The available evidence supports a reproducible and auditable protocol as one that aligns data collection, scene reset, policy execution, result logging, and task-factor analysis, with automated success classification and reset policies providing concrete mechanisms for repeatable trials. Repetition alone is therefore not the whole protocol: the conditions before execution, the decision about success, and the recorded outcome must also be part of the evaluation design.
UMI-Bench 1.0 applies this full-chain approach to tabletop manipulation with Universal Manipulation Interface-style policies. Its authors argue that these policies must be evaluated as a coupled system because their performance depends on wrist-view observations, action representation, data collection, and physical deployment. They present the benchmark as a local-first, standardized testbed for measuring generalization to physical manipulation, but the supplied abstract does not independently confirm that claim.
AutoEval supplies concrete mechanisms. Its project description says robust success classifiers judge outcomes while reset policies fine-tuned from foundation models prepare subsequent evaluations, enabling autonomous real-world operation 24/7 with minimal human intervention. The description also reports that its results correspond closely to ground-truth evaluations run by humans, although the supplied excerpt gives no tasks, comparison metric, or numerical agreement result.
These excerpts establish the main components of a reproducible and auditable evaluation architecture, but not a complete implementation recipe. They do not specify trial counts, reset tolerances, hardware controls, task definitions, success thresholds, logging fields, or a reporting format. A usable protocol would therefore need those operational details before another team could reproduce the evaluation or audit individual results; the present evidence supports the structure and reported mechanisms, not their independent validation.