Recurrent models typically use a fixed-size memory, which means they cannot grow their storage during inference. That constraint is not automatically a barrier to recall, however. A new study shows that whether a model can retrieve learned associations from a fixed matrix state depends heavily on the task it is given and how it is trained.
The authors examine a small variant of the DeltaNet architecture, adding fixed token-specific key biases. The model is trained to remember 32 new associations. The results indicate that successful recall is not a simple property of the memory mechanism itself; it emerges from the interaction between the task and the training regime.
This work is a reminder that constant-memory recall is possible, but it is not guaranteed. The conditions under which it works—and fails—are the real subject of the study.