Looped transformers reuse the same layer multiple times, which lets inference compute be adjusted without changing the model. Adaptive-depth methods build on this by training the model to work across several loop depths, so a single model can trade compute for accuracy at inference time.
According to the arXiv abstract, recent adaptive-depth methods train the model across multiple loop depths from the beginning of training. The paper's title, "Adaptive-Depth Is a Finishing Stage," signals a different view: that adaptive-depth training belongs at the end of the training pipeline, as a finishing stage, rather than as a from-scratch training regime.
The abstract is brief, so the full experimental evidence is not available from this source alone. The core claim, however, is clear: the placement of adaptive-depth training in the pipeline matters, and the authors position it as a final stage.