Using evolutionary information for label smoothing during inverse folding training leads to improved fitness prediction

Summary

Using evolutionary information for label smoothing when training inverse folding models improves variant effect prediction. Zhou et al. used the BLOSUM62 matrix, whereas Gong et al. (1) used PSSMs. By contrast, Dauparas et al. used uniform smoothing (2,3).

Details

Zhou et al. use a temperature value $t$ to balance the contribution of the WT amino acid and the BLOSUM62 matrix - the figure below defines this as ${\mathbf{B}}^{\prime }=\sigma (\mathbf{B}{)}^t$ where ${\mathbf{B}}^{\prime }$ and $\mathbf{B}$ are the target matrices and BLOSUM matrices, respectively, and $\sigma$ is a nonlinear operator. (2)

In contrast, Gong et al. (1) calculated propensities from MSAs.

Figures

Figure S7 from (2)

Method	T2837 ρ↑	T2837 AUC↑	S669 ρ↑	S669 AUC↑	S-Sym ρ↑	S-Sym AUC↑	Myolobin ρ↑	Myolobin AUC↑	FireProtDB ρ↑	FireProtDB AUC↑	Gβ1 ρ↑	Gβ1 AUC↑	T2837 Reverse ρ↑	T2837 Reverse AUC↑
# Mutations	2837		669		342		134		1764		935		2837
RaSP* (Blaabjerg et al., 2023)	0.58	0.61	0.39	0.69	0.64	0.73	0.68	0.75	0.56	0.71	0.72	0.66	0.23	0.59
ThermoMPNN* (Dieckhaus et al., 2023)	0.55	0.78	0.39	0.68	0.66	0.82	0.58	0.77	0.57	0.75	0.65	0.78	0.43	0.71
Prostata-IFML (Diaz et al., 2023)	0.53	0.75	0.49	0.76	0.55	0.75	0.54	0.67	-	-	0.66	0.82	0.52	0.75
Stability Oracle (Diaz et al., 2023)	0.59	0.81	0.52	0.74	0.72	0.87	0.68	0.81	0.61	0.79	0.71	0.82	0.59	0.81
ESM2* (Lin et al., 2023)	0.28	0.60	0.04	0.50	0.26	0.56	0.15	0.57	0.25	0.57	0.25	0.63	0.28	0.60
ProteinMPNN* (Dauparas et al., 2022)	0.36	0.70	0.25	0.59	0.32	0.64	0.35	0.66	0.31	0.70	0.35	0.67	0.36	0.70
MutComputeXGT (Diaz et al., 2023)	0.34	0.68	0.27	0.57	0.38	0.72	0.37	0.72	0.30	0.69	0.34	0.66	0.34	0.68
MutComputeXGT w/ MSA soft-label (Ours)	0.37	0.70	0.30	0.59	0.48	0.75	0.45	0.75	0.36	0.71	0.41	0.69	0.37	0.70
MutRank (Ours)	0.51	0.78	0.40	0.67	0.62	0.84	0.68	0.84	0.51	0.77	0.62	0.77	0.51	0.78
SSL Improvement ↑	42%	11%	48%	14%	63%	17%	83%	17%	65%	10%	77%	15%	42%	11%
Supervised Fine-Tuning Gap ↓	14%	4%	23%	9%	14%	3%	0%	-4%	16%	3%	13%	3%	14%	4%

Table 3 from (1)

1.

Gong C, Klivans A, Loy JM, Chen T, Liu Q, Diaz DJ. Evolution-Inspired Loss Functions for Protein Representation Learning. In: Proceedings of the 41st International Conference on Machine Learning. PMLR; 2024. p. 15893–906. (Proceedings of Machine Learning Research; vol. 235). Available from: https://proceedings.mlr.press/v235/gong24e.html

2.

Zhou B, Zheng L, Wu B, Tan Y, Lv O, Yi K, et al. Protein Engineering with Lightweight Graph Denoising Neural Networks. Journal of Chemical Information and Modeling. 2024;64(9):3650–61. Available from: https://doi.org/10.1021/acs.jcim.4c00036

3.

Dauparas J, Anishchenko I, Bennett N, Bai H, Ragotte RJ, Milles LF, et al. Robust deep learning–based protein sequence design using ProteinMPNN. Science. 2022;378(6615):49–56. Available from: https://doi.org/10.1126/science.add2187