Contrastive fine-tuning PLMs on inverse folding embeddings of experimental structures but not computational models improves downstream tasks

Summary

Contrastive fine-tuning of protein language models outputs using embeddings from inverse folding models improves downstream classification tasks (1–2). (1) used contrastive learning to fine-tune AntiBERTa2, AntiBERTy, and ESM2 to match those of ESM-IF, improving antigen classification; importantly, this only worked on experimental structures, and not computational models. (2,3) used a similar student-teacher training scheme to fine-tune ESM on structural data, and they also relied on contrastive learning, but they did not train on computational models. (3) found that family-level classification was the only case where default ESM outperformed the contrastive learning-improved version, and that the use of low-rank adaptation sometimes but not always led to improvements in prediction.

Figures

Ref (1)

See also

• Training inverse folding and diffusion models exclusively on predicted protein structures worsens performance due to how locally perfect they are

1.

Barton J, Galson JD, Leem J. Enhancing Antibody Language Models with Structural Information. openRxiv; 2024. Available from: https://doi.org/10.1101/2023.12.12.569610

2.

Liu Y, Nie Z, Chen J, Zheng X, Fu J, Liu Z, et al. Interpretable antibody-antigen interaction prediction by introducing route and priors guidance. openRxiv; 2024. Available from: https://doi.org/10.1101/2024.03.09.584264

3.

Wang D, Pourmirzaei M, Abbas UL, Zeng S, Manshour N, Esmaili F, et al. S‐PLM: Structure‐Aware Protein Language Model via Contrastive Learning Between Sequence and Structure. Advanced Science. 2024;12(5). Available from: https://doi.org/10.1002/advs.202404212