Data and benchmarks
Why is one robot benchmark score not enough?
A benchmark score answers a question set by its test: which task, starting conditions, robot and measure of success? Change the test and the same policy may face a different problem. RoboArena compares policies on physical DROID setups; a geometric stress test checks selected altered objects on physical robots.
What did the test actually compare?
Evaluators in RoboArena chose tasks for double-blind comparisons between policy pairs. The feedback included a progress score and a preference between the pair. The authors report 612 pairwise comparisons on the DROID platform.
What happens when the object changes?
The Geometric Red-Teaming study asks another question. Its researchers searched in simulation for altered object shapes that expose failures in manipulation policies, then tested selected shapes on physical robots. In an insertion test on an xARM 6, the original policy succeeded in 90% of trials with the nominal object and 22.5% with one selected altered shape. The paper reports 40 physical trials per shape for this insertion check.
The lower success rate concerns one deliberately selected object under that experiment's conditions. It is not a measured failure rate for everyday use. The authors also caution that a stress test found in simulation cannot be assumed to transfer to every physical task and object.
Read the score with its boundaries
RoboArena's pairwise preferences and the insertion study's success percentages measure different outcomes. Placing their numbers in a league table would erase the tasks, hardware and test conditions that give each number meaning.
Before trusting a claim about a robot's ability, ask what counted as success, what was varied, how many trials were run and whether the result came from simulation or hardware. A score is evidence for the question its test asked. Other questions need other tests.