Energy-based models

Energy-based models are statistical models or neural networks that attempt to learn the energy characterizing a distribution of data, rather than recovery of individual data points themselves. These are typically described as $p(A|B)=\frac{\exp \left(-E_{\theta }(A,B)/\beta \right)}{Z_{\theta ,B}}$ where $\beta$ is the inverse temperature and $Z$ is a usually unknown partition function ${\int }_{\Omega }{e}^{-\beta E_{\theta }(A,B)}$. In the context of protein structure prediction, energy-based models are useful for learning the full conformational distribution, e.g., $p(str|seq)$.

Details

Some of these methods adapt diffusion models by teaching the neural network $s_{\theta }(x,t)$ that normally learns to approximate the time-dependent score function ${\nabla }_x\log p_t(X)$ to instead directly approximate $-{\nabla }_x{E}_{\theta }(x,t)$.

ProteinEBM (1) trained using a denoising score matching framework with the following loss:

$$\frac{1}{N_r}\Vert -{\nabla }_x{E}_{\theta }(x_t,s,t)-\frac{x_t-\sqrt{{\beta }_t}{x}_0}{1-{\beta }_t}{\Vert }^2$$

$N_r$: Number of residues $x_t=\mathcal{N}(\sqrt{{\beta }_t}{x}_0,1-{\beta }_t)$: Noised version of input $x_0$

1.

Roney JP, Ou C, Ovchinnikov S. Protein Diffusion Models as Statistical Potentials. openRxiv; 2025. Available from: https://doi.org/10.64898/2025.12.09.693073