Summary
Attention matrices of PLMs correspond to 3D contacts (1,2), and they can be fine-tuned for contact prediction (3). The categorical Jacobian method (4) is another way to predict these contacts without fine-tuning. Note that this is not true of antibodies (5).
See also
1.
Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science. 2023;379(6637):1123–30. Available from: https://doi.org/10.1126/science.ade2574
2.
Rao R, Meier J, Sercu T, Ovchinnikov S, Rives A. Transformer protein language models are unsupervised structure learners. openRxiv; 2020. Available from: https://doi.org/10.1101/2020.12.15.422761
3.
Verkuil R, Kabeli O, Du Y, Wicky BIM, Milles LF, Dauparas J, et al. Language models generalize beyond natural proteins. openRxiv; 2022. Available from: https://doi.org/10.1101/2022.12.21.521521
4.
Zhang Z, Wayment-Steele HK, Brixi G, Wang H, Kern D, Ovchinnikov S. Protein language models learn evolutionary statistics of interacting sequence motifs. Proceedings of the National Academy of Sciences. 2024;121(45). Available from: https://doi.org/10.1073/pnas.2406285121
5.
Burbach SM, Briney B. Improving antibody language models with native pairing. Patterns. 2024;5(5):100967. Available from: https://doi.org/10.1016/j.patter.2024.100967