Low-rank adaptation changes the magnitude, but not directionality, of matrix operations relative to fine-tuning

Summary

Low-rank Adaptation changes the magnitude, but not direction, of matrices (1). This observation prompted the development of DoRA.

Figures

Ref (1)

1.

Liu S-Y, Wang C-Y, Yin H, Molchanov P, Wang Y-CF, Cheng K-T, et al. DoRA: Weight-Decomposed Low-Rank Adaptation. In: International Conference on Machine Learning. PMLR; 2024. p. 32100–21. Available from: https://proceedings.mlr.press/v235/liu24bn.html