Protein language models learn structure-level features, including disorder, in later layers

Summary

Protein language models learn structure-level features, including disorder, in later layers (1). This was shown using sparse autoencoders.

Figures

Ref (1)

See also

• Protein property prediction using PLMs does not benefit from scale except when predicting inferring features of either structural or sparsely populated sequence families

1.

Adams E, Bai L, Lee M, Yu Y, AlQuraishi M. From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models. openRxiv; 2025. Available from: https://doi.org/10.1101/2025.02.06.636901