Self-Adversarial Negative Sampling

Self-Adversarial Negative Sampling is a negative sampling technique used for methods like word embeddings and knowledge graph embeddings. The traditional negative sampling loss from word2vec for optimizing distance-based models be written as:

$$ L = −\log\sigma\left(\gamma − d_{r}\left(\mathbf{h}, \mathbf{t}\right)\right) − \sum^{n}_{i=1}\frac{1}{k}\log\sigma\left(d_{r}\left(\mathbf{h}^{'}_{i}, \mathbf{t}^{'}_{i}\right) - \gamma\right) $$

where $\gamma$ is a fixed margin, $\sigma$ is the sigmoid function, and $\left(\mathbf{h}^{'}_{i}, r, \mathbf{t}^{'}_{i}\right)$ is the $i$-th negative triplet.

The negative sampling loss samples the negative triplets in a uniform way. Such a uniform negative sampling suffers the problem of inefficiency since many samples are obviously false as training goes on, which does not provide any meaningful information. Therefore, the authors propose an approach called self-adversarial negative sampling, which samples negative triples according to the current embedding model. Specifically, we sample negative triples from the following distribution:

$$ p\left(h^{'}_{j}, r, t^{'}_{j} | \text{set}\left(h_{i}, r_{i}, t_{i} \right) \right) = \frac{\exp\alpha{f}_{r}\left(\mathbf{h}^{'}_{j}, \mathbf{t}^{'}_{j}\right)}{\sum_{i=1}\exp\alpha{f}_{r}\left(\mathbf{h}^{'}_{i}, \mathbf{t}^{'}_{i}\right)} $$

where $\alpha$ is the temperature of sampling. Moreover, since the sampling procedure may be costly, the authors treat the above probability as the weight of the negative sample. Therefore, the final negative sampling loss with self-adversarial training takes the following form:

$$ L = −\log\sigma\left(\gamma − d_{r}\left(\mathbf{h}, \mathbf{t}\right)\right) − \sum^{n}_{i=1}p\left(h^{'}_{i}, r, t^{'}_{i}\right)\log\sigma\left(d_{r}\left(\mathbf{h}^{'}_{i}, \mathbf{t}^{'}_{i}\right) - \gamma\right) $$