The paper, by Fady Rezk and colleagues, tackles a known gap in few-shot transfer learning: parameter-efficient fine-tuning of foundation models works well in practice, but offers no formal generalization guarantees. For high-stakes applications, such guarantees may be needed for ethical or legal reasons before deployment.
The authors propose a weight-space learning approach. First, upstream tasks are used to train a diffusion model over PEFT parameters. Then, for a downstream task, the model samples plausible PEFTs from this diffusion model and selects the one with the highest likelihood on the downstream data. This confines the hypothesis space to a finite set of PEFT samples, rather than the usual continuous space of neural network weights.
Because the hypothesis set is finite, the authors can derive tighter risk certificates. They report non-trivial generalization guarantees in the low-shot regime, where existing learning approaches yield vacuous bounds. The work is available as arXiv:2502.06970.