A new arXiv paper questions whether experience memory truly helps closed-loop LLM driving agents. The authors argue that previously published gains cannot be taken at face value, because the experiments did not control for two factors the paper identifies as decisive.
The first factor is the composition of the warmup pool—which prior driving experiences are included in memory. The second is what any injected tokens do to the prompt itself. To separate these effects, the study uses resampled warmup pools, token-matched fillers, and audited baselines.
The finding is a caution for the field: apparent improvements from experience memory may reflect prompt contamination or the luck of which experiences were selected, rather than the memory mechanism itself. The paper’s controlled setup suggests that future evaluations need to isolate these variables before claiming that memory is what makes an agent drive better.