Looped transformers reuse a shared block of layers to increase computational depth without adding trainable parameters, but training them typically relies on backpropagation through time (BPTT). To save compute, practitioners often truncate BPTT to the last few loops, even though this may weaken the benefit of looping. The authors systematically varied model size, training tokens, and forward/backward loop counts across nearly 600 runs, then fit a unified scaling law that captures looping and truncation through effective parameters.
The fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law also shows that at a fixed model size, the compute-optimal loop count and training-token budget both grow with compute.
When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and only 8% more training compute. The results suggest that looped architectures can be parameter-efficient, but only if truncation is avoided in favor of fewer, fully trained loops.