Evaluating deep learning optimizers is tricky because performance can depend more on how well hyperparameters are tuned or how closely the optimizer is co-designed with a model than on the algorithm itself. The AlgoPerf benchmark aims to separate these factors.
The updated AlgoPerf V1 results, described in a recent arXiv paper, present new findings from this effort. By controlling for tuning and co-design, the benchmark attempts to measure the true algorithmic contribution of training methods.
The significance is that easier-to-tune training algorithms could be identified more reliably, helping researchers and practitioners choose optimizers based on intrinsic merit. The paper is an update to the AlgoPerf V1 suite.