Language models can be infused with structure via low-rank adapter layers

Summary

Protein language models can be converted into inverse folding models by adding intermediate adapter layers and fine-tuning on structural data ((1), using ProGen models; (2) using ESM2-650M). This leads to improvements in sequence recovery and fitness prediction beyond what can be achieved by concatenation. Larger models maintain the improvements in perplexity. These structures can include non-protein material if the adaptor layer allows it. This is conceptually similar to LoRA. (1) validate designs in the wet lab.

Figures

Ref (1)

Ref (2)

See also

• Adding structural adaptors to language models leads to improvements in thermostability prediction compared to structure-based NNs alone

1.

Ruffolo JA, Bhatnagar A, Beazer J, Nayfach S, Russ J, Hill E, et al. Adapting protein language models for structure-conditioned design. openRxiv; 2024. Available from: https://doi.org/10.1101/2024.08.03.606485

2.

Li Z, Luo Y. Generalizable and scalable protein stability prediction with rewired protein generative models. Nature Communications. 2025;17(1). Available from: https://doi.org/10.1038/s41467-025-67609-4