Low-rank adaptation (LoRA) is a popular technique for fine-tuning large models without updating all parameters. It works by learning low-rank weight updates, which keeps training efficient. However, simply increasing the rank of these updates does not always lead to better downstream performance, because some learned directions may not be useful.
The new paper introduces ReGRA, a method for residual-gradient adaptation that aims to make low-rank fine-tuning more complementary. By focusing on residual gradients, ReGRA seeks to ensure that the added capacity from higher ranks contributes positively to the task. The approach is designed to complement existing low-rank updates rather than replace them.
The abstract notes that increasing rank expands adaptation capacity, but it does not guarantee that all learned directions improve performance. ReGRA appears to target this gap, though the full details of the method are not available in the abstract.