When Similar Robot Scenes Should Produce Different Actions

A correct action does not by itself reveal which information produced it. Counterfactual task pairs test history use by giving a policy similar current observations that require different actions because different events happened earlier, then checking whether its decisions change accordingly.

The comparison depends on both members of the pair: the present views are similar, while the task-relevant past and required response differ. Appropriate responses to both are consistent with the policy using that historical distinction rather than reacting only to the present observation. Fine-grained subtask annotations can support evaluation focused on the points where history matters. Because the admitted abstract does not describe the pair-construction controls, however, it cannot establish how fully the benchmark excludes other cues or task priors.

The authors of Grounded in Time, or GiT, describe annotated counterfactual pairs as part of a benchmark spanning biolaboratory, household, and industrial scenarios. The authors evaluated representative end-to-end vision-language-action models in simulation and on selected real-world tasks, reporting substantial room for improvement in history-dependent manipulation.

That result is an author-reported finding rather than independent confirmation, and the abstract provides no detailed scores. The defensible conclusion is therefore methodological: matched pairs can reveal whether a policy’s behavior changes with relevant prior events, while stronger claims about what the policy truly uses require controls and results not supplied in the admitted evidence.