Contrastive learning

Contrastive learning is a supervised method for driving the latent representations of data points towards or away from each other by adding custom losses.

Implementations

• The triplet margin loss used by Yu et al. (1): $$L^{TM}=||z_a-z_p|{|}^2-||z_a-z_n|{|}^2+\alpha$$ $z_a$: Enzyme embedding $z_p$: Positive case $z_n$: Negative case, selected to have EC numbers close in Euclidean space to the positive case $\alpha$: Margin; set to 1
• The supercon hard loss used by Yu et al. (1): $$L^{sup}=\sum _{e\in E}\frac{-1}{|P(E)|}\sum _{z_p\in P(e)}log\frac{\exp (z_e{z}_p/\tau )}{{\sum }_{z_a\in A(e)}\exp (z_i{z}_a/\tau )}$$ $\tau$: temperature, set to 0.1 in the paper
• Noise contrastive estimation (variant 1): $$L_B(\theta ,\gamma )=\log \sigma (s(x_i,a_{i,0}^{+};\theta ),\gamma )+\sum _{k=1}^K\log (1-\sigma (s(x_i,a_{i,k}^{-};\theta ),\gamma )$$ $x_i$: Input text $a_{i,\circ }$: Another example (either positive or negative) $s(x_i,a_{i,\circ };\theta )$ : Scoring function, usually cosine similarity, dot product, or logit “produced by input-sample matcher sub-network” (from Rethmeier and Augenstein 2021) $\sigma (z,\gamma )$: Scaling function, usually sigmoid
• Noise contrastive estimation (variant 2, ranks a single positive pair over $K$ negative pairs): Variant 2 ranks a single positive pair over $K$ negative pairs $$L_R=\log \frac{\exp (\bar{s}\ast (x_I,a_{i,0}^{+};{\theta }_{)})}{\exp (\bar{s}\ast (x_I,a_{i,0}^{+};{\theta }_{)})+{\sum }_{k=1}^K\exp (\bar{s}\ast (x_I,a_{i,k}^{-};\theta ))}$$