Transformers are often praised for their ability to process variable-length inputs and for the attention mechanism's capacity to pick out relevant features. A new paper on arXiv (2610.20337) examines how these capabilities develop during training, specifically when inputs are unbounded in length.
The study's central observation is that the output layer tends to learn its task earlier than the attention mechanism. This lag in attention learning could have implications for how transformers are trained and for understanding why they sometimes struggle with longer sequences.
The abstract also notes that the paper is situated within the broader context of transformers' expressiveness for formal languages over variable-length inputs. However, the full details of the training dynamics and the proposed explanations are not available from the abstract alone.