Large language models sometimes reply in a language the user did not expect, a failure mode known as language confusion. A new preprint investigates whether this behavior can be traced to specific internal components. The authors systematically ablated individual attention heads across 12 checkpoints from seven model families, using FLORES-200 continuations to probe the effect.
The results indicate that a single attention head can be responsible for keeping the model in the intended language. When that head is removed, the model's output can shift into an unexpected language. This suggests that language identity is not spread evenly across the network but is at least partly concentrated in a small number of units.
The paper does not claim that all language confusion stems from one head, and the abstract leaves the full experimental details unspecified. Still, the finding offers a concrete target for future interpretability work and may help explain why these models occasionally slip into the wrong tongue.