Quartz 4

Invariant point attention

Created Modified

The invariant point attention module of AlphaFold2 is a widely used SE(3)-invariant module used to process protein structures. It’s used for other tasks including error estimation (IgFold; (1)), processing of cryo-EM density (ModelAngelo, (2)), inverse folding ((3), ProteinIPMP), protein backbone design (4,5), and conformational sampling (6). Liu et al. (7) released a faster version of this called FlashIPA.

Notes

  • The invariant point attention can be retrained and used for domain segmentation (8).
  • Training RosettaFold2 with invariant point attention instead of the SE3-transformer led to “unstable training performance” (9).
  • **Each of the three attention matrices in IPA are scaled (divided) by ** **instead of the normal ** found in transformers.
  • Billera et al. (4) found that for protein backbone design, the layer norm benefited by moving to after the IPA/feed-forward layer but before the residual connection. Pre-norm led to early plateauing during training, whereas post-norm (after the residual connection) “prevented scaling the number of layers in the model”.
  • FlashIPA is a faster version of IPA:
1.
Ruffolo JA, Chu L-S, Mahajan SP, Gray JJ. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Nature Communications. 2023;14(1). Available from: https://doi.org/10.1038/s41467-023-38063-x
2.
Jamali K, Käll L, Zhang R, Brown A, Kimanius D, Scheres SHW. Automated model building and protein identification in cryo-EM maps. Nature. 2024;628(8007):450–7. Available from: https://doi.org/10.1038/s41586-024-07215-4
3.
Akpinaroglu D, Seki K, Guo A, Zhu E, Kelly MJS, Kortemme T. Structure-conditioned masked language models for protein sequence design generalize beyond the native sequence space. openRxiv; 2023. Available from: https://doi.org/10.1101/2023.12.15.571823
4.
Billera L, Oresten A, Stålmarck A, Sato K, Kaduk M, Murrell B. The Continuous Language of Protein Structure. openRxiv; 2024. Available from: https://doi.org/10.1101/2024.05.11.593685
5.
Huguet G, Vuckovic J, Fatras K, Thibodeau-Laufer E, Lemos P, Islam R, et al. Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Backbone Generation. 2024; Available from: https://arxiv.org/abs/2405.20313
6.
Lu J, Zhong B, Zhang Z, Tang J. Str2Str: A Score-based Framework for Zero-shot Protein Conformation Sampling. In: ICLR 2024. 2024. Available from: https://openreview.net/forum?id=C4BikKsgmK
7.
Liu A, Elaldi A, Franklin NT, Russell N, Atwal GS, Ban Y-EA, et al. Flash Invariant Point Attention. 2025; Available from: https://arxiv.org/abs/2505.11580
8.
Lau AM, Kandathil SM, Jones DT. Merizo: a rapid and accurate protein domain segmentation method using invariant point attention. Nature Communications. 2023;14(1). Available from: https://doi.org/10.1038/s41467-023-43934-4
9.
Baek M, Anishchenko I, Humphreys IR, Cong Q, Baker D, DiMaio F. Efficient and accurate prediction of protein structure using RoseTTAFold2. openRxiv; 2023. Available from: https://doi.org/10.1101/2023.05.24.542179

Graph View

Backlinks