Thermostability prediction from protein language model representations peaks in the last few layers

Summary

Representations from later protein language model layers are most effective for thermostability prediction. In ESMC-6B, ridge regression on raw mean-pooled representations peaked at the penultimate layer, layer 79 of 80, before dropping significantly in the final layer (1). Adams et al. likewise found that ESM-2-650M representations were most predictive of thermostability in the last layer, although they sampled at four-layer intervals rather than layer by layer (2).

See also

• Protein language models learn structure-level features, including disorder, in later layers

1.

Candido S, Hayes T, Derry A, Rao R, Lin Z, Verkuil R, et al. Language Modeling Materializes a World Model of Protein Biology. 2026 Jun; Available from: http://dx.doi.org/10.64898/2026.06.03.729735

2.

Adams E, Bai L, Lee M, Yu Y, AlQuraishi M. From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models. openRxiv; 2025. Available from: https://doi.org/10.1101/2025.02.06.636901