Robots need spatial memory to reuse past observations for localization, navigation, and question answering. Designing such memory means deciding what to retain and how to retrieve it for diverse semantic, metric, and temporal needs. The authors of this paper investigate whether coding agents can automate that design process through iterative implementation and evaluation.

Under a fixed development budget, a Designer Agent writes and revises code for memory construction and retrieval, using feedback from a fixed Query Agent. The resulting implementations are frozen before building memories from held-out observation histories, with no access to downstream questions. The systems are then evaluated across five complementary task tracks using the same Query Agent.

Across three independent design searches, Codex / GPT-6 Astra averaged 44.23 on the equally weighted five-track score, versus 29.32 for ReMEmbR, the strongest evaluated baseline supporting all five tracks—a 50.9% relative improvement. Qualitative analysis relates performance differences to evidence retention and retrieval, while also identifying grounding and evidence-use failures.

These results demonstrate the potential of coding agents to automate spatial memory design, while revealing task-dependent trade-offs in performance and computational cost. The identified failures point to areas for future refinement before such systems are deployed in real-world robotics.