A recent arXiv preprint (2610.18085) demonstrates that the relationship between a Transformer's parameters and its training dynamics is more subtle than often assumed. The authors show that two Transformers can compute exactly the same function yet follow different training trajectories.

The concrete example involves the query and key matrices in attention. If you double the query matrix and halve the key matrix, the overall function computed by the model remains unchanged. However, the gradient changes, meaning the model will update differently during training.

The paper argues that no standard optimizer is equivariant to such reparameterizations. In other words, the choice of parameterization itself can influence the path of optimization, even when the underlying function is identical. This has implications for understanding why certain initializations or scaling choices affect training.