Low-rank adaptation causes models to learn less and forget less

Summary

Low-rank Adaptation avoids catastrophic forgetting but causes models to learn less than if they were subject to full fine-tuning (1). In part this is due to strong regularization (relative to other methods such as weight decay or dropout).

See also

• Fine-tuning can be detrimental to performance

1.

Biderman D, Portes J, Gonzalez Ortiz JJ, Paul M, Greengard P, Jennings C, et al. LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research. 2024; Available from: https://openreview.net/forum?id=aloEru2qCG