Transformer Programs, introduced by Friedman et al. in 2023, constrain a Transformer's architecture so that its learned computation can be extracted as readable Python code. This makes the model's internal logic inspectable, a step beyond treating neural networks as black boxes.
A new arXiv preprint (2610.22891) examines what token accuracy hides in these programs. The abstract warns that even when the computation is exposed, accuracy metrics can obscure important aspects of the learned behavior. In other words, a model that scores well on token prediction may still have learned something different from what the code suggests.
The paper's title asks directly what accuracy hides, and the abstract suggests that interpretability and performance metrics do not always align. For researchers using Transformer Programs to understand model behavior, this is a caution that high accuracy does not guarantee that the extracted program is the whole story.