Fine-tuning a pretrained model usually involves freezing some weights to save compute and prevent overfitting. But a new arXiv paper argues that freezing weights alone is not enough: when other layers are updated, they change the inputs flowing into the frozen core, which can still degrade overall performance. The authors call this the hidden cost of naive freezing.
The paper proposes "Guarded Freezing," a method that explicitly accounts for how connectivity between layers shapes the fine-tuning process. Rather than treating frozen layers as isolated, it considers how updates elsewhere propagate through the network, and uses that information to decide which weights should actually stay frozen. The result is a more principled approach to preserving performance while still limiting the number of trainable parameters.
Because this is a single preprint, the details of the method and its empirical results are not yet available in the abstract. Still, the core insight—that freezing is a structural decision, not just a weight-level one—could have practical implications for efficiently adapting large models. Further reading of the full paper would be needed to assess how well Guarded Freezing works across different architectures and tasks.