Diffusion models

Diffusion models are generative models whose outputs are generated by iteratively denoising Gaussian noise. During inference, the reverse process is executed, whereas during training, the forward process is carried out (converting data to noise). This process can be guided at inference time to respect specific geometric constraints.

Details

Diffusion models try to recover samples ${\mathbf{X}}_0$ drawn from $p_{data}$ from Gaussian noise ${\mathbf{X}}_T$ by iteratively denoising across $T$ time steps. The forward process is defined as

$B_t$: Standard Weiner process (by which noise is added)

Typically these drift and diffusion functions are used:

The reverse process:

The scoring function is approximated by a neural network, $s_{\theta }(x,t)$.

Diffusion models can also be zero-shot classifiers (1).

Diffusion models have been combined with Replica-exchange molecular dynamics (2).

Types of sequence-based diffusion

For protein sequences, which are fundamentally discrete, Yang et al (3) describe three approaches to generation using diffusion models:

• Diffusion in pre-trained latent space (e.g., continuous diffusion)
• Diffusion in discrete space with uniform noise matrices
• Diffusion in discrete space via absorbing matrices, e.g., one-at-a-time masking/unmasking of individual tokens