How learning works
How can we tell whether a robot has learned a skill?
To assess a claim that it learned a skill, look at how success was defined, what was held out of training, and whether the result came from a physical test or a simulation.
Define the task before judging the result
The AutoEval study illustrates this in a physical evaluation system. It defines a task success check, repeats trials from a specified range of initial states and resets the scene between attempts. Its success detector and reset policy are learned components whose mistakes can affect the reported result. An automated score still depends on whether the evaluation itself worked as intended.
Test something the robot has not practiced
The BC-Z study tests a language-conditioned policy on manipulation tasks held out of training. It reports some success on many of those tasks, with substantial variation in performance. The result concerns the objects, commands and robot setup in that study. It does not establish performance on arbitrary new household tasks.
Changing the task is only one test of generalization. A researcher might also change distractor objects, camera angles or lighting conditions. The AutoEval authors say their automated system does not yet support controlled tests across several of those robustness conditions. A report should therefore say what changed in the experiment and what remained untested.
Distinguish physical trials from recovery in simulation
Simulated trials and physical trials answer different questions. AutoEval's authors note that gaps between simulation and reality remain, and they run their evaluation on real robot systems. Those trials still cover the tasks and scenes actually set up for that system.
Another question begins after an action disrupts the task. In the RoboRecover preprint, researchers reconstruct an intermediate state by replaying actions, then test whether a policy can complete the original task from there. The original instruction and success criterion remain in force. Its RoboTwin and LIBERO scenarios are simulated, so this recovery result is separate from a physical robot's reliability.
When judging a claim about a learned skill, ask for the task and success rule, the training and test conditions, the number and range of trials, and the robot or simulator used. Ask separately whether the policy was tested after an error. A study can give good evidence for one of these questions while leaving the others open.