Loss Functions

# InfoNCE

Introduced by Oord et al. in Representation Learning with Contrastive Predictive Coding

InfoNCE, where NCE stands for Noise-Contrastive Estimation, is a type of contrastive loss function used for self-supervised learning.

Given a set $X =${$x_{1}, \dots, x_{N}$} of $N$ random samples containing one positive sample from $p\left(x_{t+k}|c_{t}\right)$ and $N − 1$ negative samples from the 'proposal' distribution $p\left(x_{t+k}\right)$, we optimize:

$$\mathcal{L}_{N} = - \mathbb{E}_{X}\left[\log\frac{f_{k}\left(x_{t+k}, c_{t}\right)}{\sum_{x_{j}\in{X}}f_{k}\left(x_{j}, c_{t}\right)}\right]$$

Optimizing this loss will result in $f_{k}\left(x_{t+k}, c_{t}\right)$ estimating the density ratio, which is:

$$f_{k}\left(x_{t+k}, c_{t}\right) \propto \frac{p\left(x_{t+k}|c_{t}\right)}{p\left(x_{t+k}\right)}$$

#### Papers

Paper Code Results Date Stars

#### Components

Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign