Data and benchmarks
How a Benchmark Can Reveal Dexterous Transfer Beyond Training Examples
A credible evaluation varies hands, arms, objects, relations, skill combinations, and deformable materials, then diagnoses exactly where an attempted transfer fails.
To show generalization beyond demonstrations, a dexterous-robot benchmark should withhold target-task demonstrations and test transfer across robot embodiments, assets, relations, recomposed skills, and deformable-object interactions. Each factor needs to be identified so that success on a novel object, a new arm-and-hand pairing, or a new combination of familiar skills is not confused with execution under demonstration-supported conditions.
HandsOn provides one example of this separation. Its authors describe a simulation platform with five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets placed in randomized scenes. The benchmark pairs 18,000 curated human demonstrations with 66 human-verified tasks. Core-50 covers demonstration-based learning and execution, while Transfer-16 supplies no training demonstrations for its target tasks and evaluates asset and relation transfer, skill composition, and deformable manipulation.
A benchmark also needs more than one final success percentage. HandsOn reports stage-level diagnostics for trained policies, distinguishing failures to establish contact, coordinate actions, preserve an already achieved goal, and sequence actions in composed tasks. These categories reveal which part of an attempted transfer broke down instead of collapsing every failure into the same result.
Reported scores should remain tied to their evaluation regime. The HandsOn authors report 29.1% success on Core-50 for the best trained multi-task policy. Without benchmark finetuning, GPT-6 using move_eef for palm-and-finger control achieved 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. Those figures are simulation results from the paper, while its real-robot rollouts are described only as qualitative illustrations of simulation-to-reality transfer. The supported conclusion is therefore limited: a benchmark can demonstrate transfer across its named held-out factors, but qualitative robot trials do not establish quantitative real-world generalization.