BERT-based inverse folding models generate overly repetitive sequences

Summary

BERT-based inverse folding models generate overly repetitive sequences (1). This can be avoided by retraining the models with custom losses that look at overall sequence composition.

Details

The loss function used to improve repetitive sequences is as follows:

$${\mathcal{L}}_{comp}(\theta )=\mathbb{E}\left(d(\mathbf{s},\hat{\mathbf{s}})\right)$$ $$d(\mathbf{s},\hat{\mathbf{s}})=-cos(\mathbf{y},\hat{\mathbf{y}})=-\frac{{\mathbf{y}}^T\hat{\mathbf{y}}}{\Vert \mathbf{y}\Vert \Vert \hat{\mathbf{y}}\Vert }$$

Figures

Ref (1)

1.

Kim N, Kim M, Ahn S, Park J. Decoupled Sequence and Structure Generation for Realistic Antibody Design. Transactions on Machine Learning Research. 2025; Available from: https://openreview.net/forum?id=CTkABQvnkm