Low-rank optimizers have become a popular way to reduce the memory and compute cost of training large language models. A family of methods, starting with GaLore, works by projecting gradients into a low-dimensional subspace at each step, then mapping the update back to the full model.
The new paper introduces Ori, a drift-aware streaming k-PCA method designed for this setting. The key issue it addresses is that the subspace itself can drift over time, which can undermine the projection and slow convergence. Ori is intended to track the subspace more reliably.
The abstract is brief, so details on the algorithm and experimental results are not yet available in the summary. The contribution appears to be a more stable way to maintain the low-rank structure that these optimizers rely on.