Overtrained language models are more difficult to fine-tune

Summary

Overtrained language models are more difficult to fine-tune (1).

1.

Springer JM, Goyal S, Wen K, Kumar T, Yue X, Malladi S, et al. Overtrained Language Models Are Harder to Fine-Tune. In: International Conference on Machine Learning. PMLR; 2025. p. 56719–89. Available from: https://proceedings.mlr.press/v267/springer25a.html