A new preprint on arXiv takes aim at how agent memory is measured. The authors note that most benchmarks evaluate whether an agent's responses reflect correct knowledge, using downstream question-answering accuracy as a stand-in for memory quality. They argue this paradigm is incomplete, because it conflates memory with other aspects of agent performance.
The abstract, which is truncated in the source, does not yet spell out the proposed alternative in detail. What is clear is the central critique: QA accuracy alone cannot tell us whether an agent truly remembers, or is merely producing correct answers through other means. The paper urges the community to rethink evaluation strategies for agent memory.
For now, the contribution is a cautionary one. As agents become more capable, benchmarks that separate memory from reasoning and other skills will likely be needed. This preprint adds to that conversation by questioning a widely used proxy.