A new paper on arXiv examines what happens when transformer networks generate text over long periods. The authors show that even a very small model—a three-token, single-layer attention-only transformer—can fail to converge. This challenges the assumption that such architectures will settle into stable behavior as they generate more tokens.
The result is notable because it applies to a minimal configuration, suggesting that non-convergence may be more common than previously expected. The paper then moves on to study a broader class of transformers, though the abstract cuts off before specifying which class or what conclusions are drawn. Readers interested in the full analysis will need to consult the complete manuscript.
As an early result, the finding highlights the need for careful study of transformer dynamics over extended sequences, especially as these models are increasingly used for long-form generation. The authors do not yet provide a full characterization, but the initial example is enough to show that convergence is not guaranteed even in simple cases.