Most studies of language model development ask a simple question: when does the model start producing the right answer? According to a new arXiv paper, that approach conflates two separate things — the model's internal knowledge and the readout that converts it into token probabilities. The readout, also called the unembedding matrix, is the final linear transformation before prediction, and the authors argue it deserves direct study.

The abstract, which is truncated, suggests that tracking the dynamics of this readout during pretraining could give a clearer picture of when capabilities emerge. Rather than treating the output layer as a fixed decoder, the paper appears to treat it as an active component of learning.

The significance is methodological: if the readout changes over time, then accuracy-based developmental curves may be misleading. The paper is a cross-listed arXiv preprint, and the full details are not yet available in the abstract, but the framing points toward a new way of interpreting pretraining dynamics.