Understanding Self-supervised Learning via Information Bottleneck Principle
Self-supervised learning alleviates the massive demands for annotations in deep learning, and recent advances are mainly dominated by contrastive learning. Existed contrastive learning methods narrows the distance between positive pair while neglect the redundancy shared by positive pairs. To address this issue, we introduce the information bottleneck principle and propose the Self-supervised Variational Information Bottleneck (SVIB) learning framework. Specifically, We apply Gaussian Mixture Model (GMM) as the parametric approximate posterior distribution to the real feature distribution and introduce the categorical latent variable. Features from different augmentations are forced to infer the other one and the latent variable together. Then, we propose variational information bottleneck as our objective, which is composed of two parts. The first is maximizing the mutual information between the inferred feature and the latent variable. The second is minimizing the mutual information between the other feature and the latent variable. Compare to previous works, SVIB provides the self-supervised learning field with a novel perspective from the variational information bottleneck, while also highlighting a long-neglected issue. Experiments show that SVIB outperforms current SOTA methods in multiple benchmarks.
PDF Abstract
ImageNet
MS COCO
