Representation geometry

How similarities, clusters, distances, and manifolds in model representations relate to sequence, structure, or function. Practical feature extraction is a separate inference topic.

Quartz 4

Explorer

Representation geometry

Protein language models learn structure-level features, including disorder, in later layers

MSA-conditioned AlphaFold3 representations compress as the Pairformer proceeds

Removing MSAs collapses AlphaFold3's latent space

Antibody language models are able to distinguish correctly and incorrectly paired sequences, but are sensitive to scale and training dataset size

CDR representations segregate into distinct clusters

Contrastive learning improves segregation of functional clusters

Distance between PLM representations of two proteins correlates with functional dissimilarity

Distance between averaged PLM embeddings does not correlate with structural difference

Encoder-decoder PLM embeddings are better able to distinguish nearest neighbors than encoder-only PLMs

High-confidence predictions from protein language models co-cluster together in embedding space and correlate with performance on variant effect prediction tasks

PLM-derived antibody representations can distinguish engineered from human-derived Abs

The latent space of ESMFold is dicontinuous, but can be smoothened by channel compression

Contrastive learning on whole structures leads to learning of distinct substructures

Dimensionality reduction