Contrastive Multiview Coding

ECCV 2020  ·  Yonglong Tian, Dilip Krishnan, Phillip Isola ·

Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We investigate the classic hypothesis that a powerful representation is one that models view-invariant factors. We study this hypothesis under the framework of multiview contrastive learning, where we learn a representation that aims to maximize mutual information between different views of the same scene but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. We analyze key properties of the approach that make it work, finding that the contrastive loss outperforms a popular alternative based on cross-view prediction, and that the more views we learn from, the better the resulting representation captures underlying scene semantics. Our approach achieves state-of-the-art results on image and video unsupervised learning benchmarks. Code is released at:

PDF Abstract ECCV 2020 PDF ECCV 2020 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Self-Supervised Image Classification ImageNet CMC (ResNet-50 x2) Top 1 Accuracy 70.6% # 96
Top 5 Accuracy 89.7% # 25
Number of Params 188M # 27
Self-Supervised Image Classification ImageNet CMC (ResNet-50) Top 1 Accuracy 66.2% # 103
Top 5 Accuracy 87.0% # 27
Number of Params 47M # 45
Self-Supervised Image Classification ImageNet CMC (ResNet-101)-deprecated Top 1 Accuracy 65.0% # 107
Top 5 Accuracy 86.0% # 28

Results from Other Papers

Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Self-Supervised Action Recognition UCF101 Contrastive Multiview Coding (CaffeNet x2) 3-fold Accuracy 59.1 # 48
Pre-Training Dataset UCF101 # 1
Frozen false # 1