Quartz 4

❯

❯

❯

Pretraining and scaling

Pretraining and scaling

Created Sep 26, 2026Modified Sep 07, 2026

Pretraining objectives, masking, model/data/compute scaling, and foundation-model learning dynamics. Dataset composition is covered by evidence/datasets when central.

39 items with this tag.

Aug 25, 2026
Pretraining performance does not capture effectiveness on downstream tasks
Aug 19, 2026
Unconditional design does not consistently improve with model scale
- design/sequence-generation
- training/pretraining-and-scaling
Jul 28, 2026
MSA Transformer
- training/pretraining-and-scaling
- model-design/architectures
Jul 28, 2026
PLMs are biased by uneven distribution of sequence data in datasets such as UniRef and UniProt
- evidence/datasets
- training/pretraining-and-scaling
Jul 28, 2026
Sequence-only protein language models implicitly cluster proteins at fineness levels that increase with size
- model-analysis/interpretability
- training/pretraining-and-scaling
Jul 28, 2026
Training antibody language models on normalized mutation frequencies improves zero-shot expression prediction
Jul 22, 2026
Protein property prediction using PLMs does not benefit from scale except when predicting inferring features of either structural or sparsely populated sequence families
- training/pretraining-and-scaling
- prediction/variant-effects
Jul 22, 2026
Quanta hypothesis
- training/pretraining-and-scaling
- model-analysis/interpretability
Jul 20, 2026
Explicit germline encoding improves prediction of non-germline CDRH3 residues
- training/pretraining-and-scaling
- antibodies/repertoires
Jul 17, 2026
Antibody-antigen complex prediction scales with training data and compute
Apr 21, 2026
AlphaFold3 learns intramolecular interactions faster than intermolecular interactions
- training/pretraining-and-scaling
- prediction/complexes
Apr 21, 2026
Alternate sequence clustering schemes outperform uniform sampling when training protein language models
- evidence/datasets
- training/pretraining-and-scaling
Apr 21, 2026
Alternative noise schedules improve training of multimodal PLMs
- training/pretraining-and-scaling
- model-design/multimodal
Apr 21, 2026
BERT models trained on sequence clusters outperform those trained on all data
- evidence/datasets
- training/pretraining-and-scaling
Apr 21, 2026
Distillation
- evidence/datasets
- training/pretraining-and-scaling
Apr 21, 2026
ESM
- training/pretraining-and-scaling
- inference/feature-extraction
Apr 21, 2026
IPA-based structure prediction neural networks train sequentially along three spatial axes
- training/pretraining-and-scaling
- prediction/structure
Apr 21, 2026
Including sequences from multiplexed ancestral sequence reconstruction improves PLM training
Apr 21, 2026
Larger PLMs are better at homolog detection
- evolution/homology
- training/pretraining-and-scaling
Apr 21, 2026
Larger PLMs are better at thermostability prediction
- prediction/stability-expression
- training/pretraining-and-scaling
Apr 21, 2026
Larger PLMs generate more novel sequences from more sparsely populated protein families
- design/sequence-generation
- training/pretraining-and-scaling
Apr 21, 2026
Masked PLMs are more sensitive to training imbalances than autoregressive PLMs
- evidence/datasets
- training/pretraining-and-scaling
Apr 21, 2026
Optimal scaling of antibody LMs improves prediction of masked CDRH3 residues but not framework residues
- training/pretraining-and-scaling
- antibodies/architecture
Apr 21, 2026
Paired antibody LMs outperform equivalent unpaired LMs on sequence recovery for various regions
Apr 21, 2026
ProGen
- training/pretraining-and-scaling
- design/sequence-generation
Apr 21, 2026
Protein language models have a lower bound of training loss regardless of size
- training/pretraining-and-scaling
Apr 21, 2026
Protein language models make equally effective predictions when trained on individual proteins or protein families
- training/pretraining-and-scaling
- prediction/variant-effects
Apr 21, 2026
Protein models designed using inverse folding can be used to supplement training DBs for PLMs and structure prediction models
- evidence/datasets
- training/pretraining-and-scaling
Apr 21, 2026
Subnetworks within protein language models encode specific protein families
- model-analysis/interpretability
- training/pretraining-and-scaling
Apr 21, 2026
Training on newly updated databases doesn't guarantee better performance
- evidence/datasets
- training/pretraining-and-scaling
Apr 21, 2026
Tranception
- prediction/variant-effects
- training/pretraining-and-scaling
Apr 21, 2026
Correlation between sequence log-likelihood and variant effect prediction performance breaks down as PLMs get larger
Apr 21, 2026
Grokking
- training/pretraining-and-scaling
- training/fine-tuning
Apr 21, 2026
Including optimal growth temperature during pre-training of PLMs improves prediction and design of thermostability
Apr 21, 2026
On fixed compute budgets, mixture-of-experts models outperform dense models
- model-design/architectures
- training/pretraining-and-scaling
Apr 21, 2026
Overtrained language models are more difficult to fine-tune
- training/fine-tuning
- training/pretraining-and-scaling
Apr 21, 2026
Scaling hypothesis
- training/pretraining-and-scaling
Apr 20, 2026
Protein language models
- training/pretraining-and-scaling
- inference/feature-extraction
Apr 17, 2026
BERT
- model-design/architectures
- training/pretraining-and-scaling

Created with Quartz v4.5.2 © 2026

GitHub
Discord Community