A new arXiv paper from Islamov, Machacek, Lucchi, Silveti-Falls, Gorbunov, and Cevher analyzes how batch size affects stochastic conditional gradient methods, focusing on momentum-based variants such as Scion. The authors derive an analysis that explicitly links stepsize, batch size, and stochastic noise under a μ-Kurdyka-Łojasiewicz condition, a common assumption in optimization theory.

The central result is regime-dependent: increasing batch size initially improves optimization accuracy, but beyond a critical threshold the benefits saturate and can eventually degrade performance when the total token budget is fixed. The theory also predicts the magnitude of the optimal stepsize, which the authors say matches empirical practices observed in large-scale training.

Based on these insights, the paper proposes principled guidelines for choosing batch size and stepsize, including an adaptive strategy that increases batch size and sequence length during training while maintaining convergence guarantees. Experiments on NanoGPT are consistent with the theoretical predictions and show the expected scaling regimes.

This work provides a theoretical framework for understanding batch size scaling in stochastic conditional gradient methods and offers practical guidance for designing efficient training schedules in large-scale optimization. Since this is a single source, no conflicting findings are noted.