A new preprint on arXiv argues that when researchers claim a robot uses memory, the standard evidence may not support that claim. The authors of "Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems" contend that success-rate ablations—where memory is removed and performance is measured—cannot by themselves show that task-relevant memories are being used.

The problem, according to the abstract, is that such ablations change both the information available to the system and the computation that is performed. Aggregate success rates conflate the two, so a drop in performance might stem from altered processing rather than from loss of specific memories. The paper therefore calls for an interventional approach that isolates the role of memory more directly.

The abstract is truncated, so details of the proposed interventional method are not available from the source. What is clear is that the authors see success-rate ablations as necessary but insufficient, and they argue for evaluations that can attribute behavior to specific stored facts.