Adapting large pretrained models to multiple downstream tasks usually forces a trade-off: full fine-tuning is expensive, while keeping separate task-specific models multiplies storage and maintenance overhead. A new arXiv paper, PGSB, proposes a middle path by merging LoRA adapters—small, trainable modules added to a frozen base model—into a single set of weights.
The method, called Pretrained-Guided Shared Basis, uses the pretrained model itself as a guide for constructing a shared basis across tasks. Instead of treating each LoRA adapter independently, PGSB aligns them in a common space derived from the base model's structure, which should make merging more coherent and less lossy.
The abstract is truncated, so the paper's full experimental results are not available in the source. What is clear is the motivation: reduce the cost of multi-task adaptation without giving up the efficiency of parameter-efficient fine-tuning. The authors position PGSB as a direct response to the storage and compute burden of maintaining many task-specific models.