Transformer

Transformers are a type of neural network widely used in language modeling. Most Protein language models and protein folding methods use transformers in some capacity.

Types

• BERT: The original encoder-only approach
• DeBERTa: A modified version with Disentangled attention
• GPT: An autoregressive decoder-only approach

Architecture

• Positional encoding
• Self-attention
• Layer normalization
• Residual connections

Extensions

• Mixture-of-depths
• Mixture-of-experts
• Low-rank Adaptation
• Model merging
• Weight averaging
• Grokking

See also

• Scaling hypothesis
• Catastrophic forgetting