Learning methods
A Robot Memory That Saves What the Present Cannot Explain
TTT-RM pairs a limited memory bank with online parameter updates aimed at the task-relevant parts of history missing from the policy’s current context.
Long manipulation sequences create a trade-off: earlier events may still matter after they disappear from the robot’s observations, but retaining every observation becomes increasingly costly. Residual memory helps by training a robot policy’s fast weights to preserve task-relevant information that its current observation and retrieved bounded memory cannot reconstruct.
TTT-RM complements a bounded memory bank with fast weights that encode complementary historical information over time. A bounded memory bank holds a limited selection of past information, while a history decoder tries to reconstruct historical representations from the current observation and whatever the bank retrieves. The difference between the history and that reconstruction is the residual: information the available context fails to explain. The method uses this residual as the learning target for Test-Time Training, optimizing slow weights so that fast weights updated online can encode the complementary history. The policy then queries that fast-weight state to obtain a residual-memory representation that conditions its actions.
In their preprint, the authors report improvements across several memory designs on memory-intensive simulation benchmarks and real-world tasks, as well as sustained execution of a three-minute, eight-stage physical stowing task. These are author-reported results from one preprint rather than independent confirmation, and the available abstract does not give task-level metrics. The supported contribution is therefore the mechanism: dividing memory between a finite store for retrievable observations and an adaptive state focused on relevant information that would otherwise be lost.