Data and benchmarks
RobotAgentBench Tests the Autonomy Loop Around Manipulation
The benchmark probes whether household robot agents can discover tasks, interpret context, act, recover from failures and integrate those skills around a user’s broader objective.
RobotAgentBench shifts evaluation from a robot’s ability to carry out a specified action toward its ability to manage an open-ended household objective. Beyond manipulation success, RobotAgentBench measures task discovery, contextual reasoning, failure recovery and robustness, then tests at scenario level whether an agent can combine these abilities with physical execution to fulfill a high-level user need.
The benchmark authors describe an agent-assisted, human-in-the-loop real-to-simulation pipeline used to construct 50 household layouts with interactive objects and physically grounded tasks. The resulting test set encompasses 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks and 4 open-ended scenarios. Standalone vision-language-action and world-action models can therefore be compared with robot-agent architectures under the same planning, execution and verification framework.
In experiments reported by the authors, the benchmark exposes substantial gaps between manipulation and broader autonomy while indicating advantages for agentic architectures and recursive self-improvement. These are claims from the benchmark’s primary preprint abstract, not independent confirmation; the admitted evidence does not specify the per-dimension scoring rules or numerical results. The benchmark’s central contribution is thus its evaluation of the full autonomy loop: finding a task, interpreting its context, acting, checking progress and adapting when execution fails.