Why Breadth Matters in Robot Training Data

Dataset diversity can improve a robot policy’s ability to generalize by exposing it to variation across tasks, environments, robot arms, and grippers, making performance less dependent on the combinations most common in training. Here, generalization means applying learned behavior under conditions that differ from those represented most heavily in the demonstrations.

Raw dataset size measures the amount of experience available, but it does not establish how broadly that experience is distributed. The Open X-Embodiment dataset aggregates demonstrations from more than 60 robot datasets, yet the OXE-AugE authors report that its four most represented robot types account for over 85% of the real data. A collection can therefore be large while remaining concentrated around a small number of robot-scene combinations.

BridgeData V2 illustrates diversity across tasks and settings. Its authors describe 53,896 trajectories collected with a publicly available low-cost robot across 24 environments. They evaluated six imitation-learning and offline reinforcement-learning methods on tasks requiring different amounts of generalization, reporting that the methods succeeded and improved with more data and higher-capacity models. They also state that skills learned from the dataset generalized across institutions. The admitted abstract, however, does not provide the individual success rates or enough experimental detail to separate the effect of environmental diversity from the effects of dataset size and model capacity.

OXE-AugE examines hardware coverage by augmenting OXE with nine robot embodiments, meaning combinations of robot arms and grippers. The resulting dataset contains over 4.4 million trajectories, more than triple the size of the original OXE. Its authors report benefits on augmented robots, unseen robots, and original robots under distribution shifts. In physical experiments across four real-world manipulation tasks, they say that fine-tuning OpenVLA and π₀ on OXE-AugE improved success rates by 24–45% on previously unseen robot-gripper combinations.

That result supports a link between embodiment coverage and cross-robot transfer, but it does not isolate diversity as the sole cause because OXE-AugE increased both the number of embodiments and the total number of trajectories. The physical evaluation also covered only four tasks, so it cannot establish the same gain across robot manipulation generally. Taken together, the studies indicate that useful diversity is coverage of the particular changes a policy may face, while trajectory count alone is an incomplete measure of that coverage.