Loss Functions

InfoNCE, where NCE stands for Noise-Contrastive Estimation, is a type of contrastive loss function used for self-supervised learning.

Given a set $X = ${$x_{1}, \dots, x_{N}$} of $N$ random samples containing one positive sample from $p\left(x_{t+k}|c_{t}\right)$ and $N − 1$ negative samples from the 'proposal' distribution $p\left(x_{t+k}\right)$, we optimize:

$$ \mathcal{L}_{N} = - \mathbb{E}_{X}\left[\log\frac{f_{k}\left(x_{t+k}, c_{t}\right)}{\sum_{x_{j}\in{X}}f_{k}\left(x_{j}, c_{t}\right)}\right] $$

Optimizing this loss will result in $f_{k}\left(x_{t+k}, c_{t}\right)$ estimating the density ratio, which is:

$$ f_{k}\left(x_{t+k}, c_{t}\right) \propto \frac{p\left(x_{t+k}|c_{t}\right)}{p\left(x_{t+k}\right)} $$

Source: Representation Learning with Contrastive Predictive Coding


Paper Code Results Date Stars


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign