Zeroth-order (ZO) optimization is an attractive route for fine-tuning large language models because it avoids storing full gradients, cutting memory use. But the approach struggles with high variance when the parameter space is very high-dimensional, which is typical for LLMs.

The new arXiv preprint (2610.15225) tackles this by proposing to perform ZO optimization within a learnable low-rank subspace. Rather than updating all parameters, the method restricts changes to a lower-dimensional space that is itself learned, potentially reducing variance and improving convergence.

The abstract is brief, so details of the subspace learning procedure and experimental results are not yet available from this source. The work points to a promising direction for making memory-efficient fine-tuning more stable.