Long-horizon failures in language models are usually measured by whether the final task succeeds or by how many steps the model can handle. A new arXiv preprint argues that this misses the structure of failure. Instead, the authors propose scoring a 400-step ledger task step by step, separating three components: duplicate recognition, local processing, and counter tracking.
Each step in the task is easy in isolation, so the difficulty comes from maintaining accuracy across the full sequence. By tracking the three components independently, the authors aim to identify which specific skill degrades as the horizon grows, rather than treating failure as a single event.
The preprint is new and not yet peer-reviewed, but the approach offers a more granular way to study reliability. If the decomposition holds, it could help researchers target the exact mechanism behind long-horizon errors in language models.