How learning works
Anatomy of a Robot Demonstration Dataset
The documented LeRobotDataset v3 and robomimic formats store observations and actions, organize records into episodes or trajectories, and include metadata describing their structure.
Beyond video, the documented examples can include camera images, robot states, recorded actions and timestamps, with episode or trajectory records and metadata describing features, shapes and task or environment context. An observation captures what was available at a step, such as a camera image or robot state; an action records the corresponding command represented in the dataset.
Observations, actions and trajectories
Robomimic’s official documentation divides observations into keys, modalities and groups. A key can represent a wrist-camera image, end-effector position or joint velocity. Modalities classify keys as low-dimensional state, RGB, depth or scan data, while groups determine which observations are passed together to a learning model.
Within a robomimic-compatible HDF5 file, each trajectory has its own demonstration group and a recorded number of state-action samples. Time-ordered fields can include actions, current and next observations, rewards, termination signals and raw MuJoCo states. Dataset-level environment arguments describe the environment or task used for collection, and optional filter keys identify subsets such as training and validation demonstrations.
Another way to package the same layers
LeRobotDataset v3 uses Apache Parquet for low-dimensional states, actions and timestamps, while camera frames are encoded in MP4. Its metadata records episode lengths and offsets, allowing rows and video frames from multiple episodes to share larger files while the public interface still provides episode-level access. A loaded sample can contain tensors for state, action, image and timestamp.
Its documented metadata also covers feature names, data types, shapes, frame rate, normalization statistics, task descriptions, codebase version and file-location templates. Robomimic specifies its own storage conventions, including channel-last (H, W, C) images stored as np.uint8 and actions normalized between -1 and 1. These details matter because a reader or training pipeline must know what each field means and how it is encoded.
What the records do not prove
These files are inputs to learning algorithms, not learned policies themselves. Recorded actions show which commands the dataset contains, while rewards and termination flags follow the meanings assigned by a particular format or environment. Their presence alone is not independent confirmation that a robot successfully completed the task.
To read a dataset description, look for its observation keys and shapes, action convention, ordering, episode boundaries, task or environment context, normalization statistics and subset labels.