Data and benchmarks
Who checks the model that judges a robot's success?
Checking the judge helps readers interpret an automated evaluation.
AutoEval's authors train success classifiers on manually labeled images, then test them on held-out images and observations collected during human control of the robot. Our reading check is to compare those checks with the evaluation setting.
Separate the action from the judgment
In AutoEval, a learned classifier judges the robot-task outcome. The classifier uses an image and a task question to predict success. A correct judgment and a successful robot action are different measurements: a judge can correctly identify a failed attempt.
Look for a check of the judge itself
AutoEval's appendix reports deploying scene-specific classifiers whose accuracy exceeds 95%. That figure concerns the classifier's judgments, not the percentage of tasks solved by the evaluated policy. The authors also report occasional scoring and reset errors, so automated results do not exactly match human-run evaluations.
Ask what happens when the scene changes
A separate AAAI study by Fiona Luo fine-tunes MiniGPT-4 for success detection on Bridge and AUTOLab UR5 robot trajectories. Its abstract reports difficulty transferring across environments, with visual variation more challenging than language variation. These findings describe that detector.
Look for differences between the judge's reported checks and the actual evaluation scene. If the paper leaves those details unclear, keep the judge's reliability as an open question when interpreting the robot's result.