Parameter-efficient fine-tuning (PEFT) lets large language models adapt to specialized tasks, but it frequently comes at the cost of the general abilities learned during pretraining. A new arXiv paper argues that the solution may lie in being more careful about which layers are adapted.
Titled "Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention," the research suggests that the location of fine-tuning within the model is a key factor. The authors propose a layer-selective approach, though the abstract does not detail the method's specifics.
Because the abstract is truncated, the full findings are not yet available. The paper is available on arXiv under identifier 2610.11620.