A new paper on arXiv challenges a core assumption behind momentum-based optimizers. Existing theory typically assumes that gradients arrive at every parameter at a roughly constant rate. In practice, heavy-tailed data distributions and modern architectures produce sparse updates, where many parameters are updated only occasionally. The authors, Katie Everett and Elliot Paquette, analyze two tractable models—least squares with sparse inputs and logistic regression with a rare class—and derive exact closed-form second-moment dynamics in the high-dimensional limit.
The key result is that the behavior of momentum is governed by the ratio of two intrinsic timescales: how long the momentum buffer survives across active updates, and how many active updates are needed to reduce the squared error. When learning is much slower than retention, the limit matches standard SGD. When learning is faster, the system becomes unstable. Only when the two timescales coincide does classical heavy-ball dynamics emerge.
The authors also find that oscillatory dynamics occur at different momentum values depending on token sparsity. This creates a spectral conflict for global momentum across token frequencies, meaning a single momentum hyperparameter cannot simultaneously match the optimal behavior for all sparsity levels. The work provides a theoretical framework for understanding why momentum may behave unexpectedly in sparse, high-dimensional settings, and it suggests that adaptive or frequency-aware momentum schemes may be needed.