How do we test whether a robot can recover after an error?

Can a policy continue its original task after an action has disrupted progress? That is the recovery question. A useful test begins from the altered state and keeps the original task criterion, so a successful clean-start run cannot stand in for the result.

Start the test where the problem begins

The RoboRecover preprint takes natural candidates from failed policy trajectories and also includes cases made by perturbing actions or by operators in the simulator. The researchers replay an action prefix to reconstruct each recovery point. A candidate policy receives the original instruction and is tested against the original success criterion, with a fresh limit on its own steps. The replayed actions do not use up that limit. This is a simulation test, not a way for a physical robot to reverse time.

The authors report 2,000 RoboRecover scenarios across RoboTwin and LIBERO: 1,000 on each simulated platform, with a fixed 800/200 train/test split per platform. The scenarios do not all come from spontaneous policy mistakes. On LIBERO, the fixed test set includes 42 cases from ordinary policy runs, 127 made with action perturbations and 31 constructed by operators. That mix matters when interpreting a score: it describes a selected set of recoverable states, not how often a robot would make each error in a home or factory.

Why initial-state success can hide weak recovery

On RoboRecover's LIBERO tests, the authors report an Initial-State Success Rate of 98.83% and an Overall Recovery Success Rate (RSR) of 48.00% for UnifoLM. For π0.5, they report 94.17% initial-state success and 64.40% overall recovery. Within this benchmark, the policy with the higher initial-state score had the lower recovery score. These numbers do not establish which policy would work best on a physical robot or why one recovered more often.

Candidate scenarios were deliberately screened before the train/test split. Four reference policies attempted each candidate state five times, and the authors retained states with between two and 18 successes out of 20. This excluded both nearly trivial and nearly unsolvable cases. The recovery percentages therefore answer a useful, narrower question about the selected benchmark. They are not a general failure rate for deployed robots.

What these studies leave open

The LIBERO-RECOVER preprint also studies recovery from simulated LIBERO states. FRBench includes both simulated and physical bimanual tasks. Their scores are not a single leaderboard.

RoboRecover's authors point to real-world systems as future work. That limit matters when reading the reported recovery rates: they describe selected simulated states, not a physical robot's reliability. A clean-start score alone does not answer how the same policy continues the task after its actions change the scene.