When a language model's ability to recall a fact declines, it is natural to ask whether the knowledge is gone or simply harder to reach. Standard test-time continual learning evaluation, which relies on current output accuracy, cannot easily answer that question. A new arXiv paper argues that this distinction matters for understanding how models learn and forget.
The paper introduces the idea of binding-specific relearning savings: a measure of how much faster a model can relearn a specific fact after it has become difficult to elicit. If relearning is quick, the underlying knowledge may still be present, just less accessible. This shifts the focus from what the model can produce at a given moment to what it can recover.
The work is positioned as a critique of current evaluation practice for parameter-updating test-time continual learning agents. By proposing a more nuanced metric, it suggests that accuracy alone may understate what a model retains. The full details of the method and results are in the preprint.