Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision
Discriminative self-supervised learning allows training models on any random group of internet images, and possibly recover salient information that helps differentiate between the images. Applied to ImageNet, this leads to object centric features that perform on par with supervised features on most object-centric downstream tasks. In this work, we question if using this ability, we can learn any salient and more representative information present in diverse unbounded set of images from across the globe. To do so, we train models on billions of random images without any data pre-processing or prior assumptions about what we want the model to learn. We scale our model size to dense 10 billion parameters to avoid underfitting on a large data size. We extensively study and validate our model performance on over 50 benchmarks including fairness, robustness to distribution shift, geographical diversity, fine grained recognition, image copy detection and many image classification datasets. The resulting model, not only captures well semantic information, it also captures information about artistic style and learns salient information such as geolocations and multilingual word embeddings based on visual content only. More importantly, we discover that such model is more robust, more fair, less harmful and less biased than supervised models or models trained on object centric datasets such as ImageNet.
PDF AbstractCode
Colab
Tasks
Action Classification
Action Recognition
Copy Detection
Domain Generalization
Fairness
Fine-Grained Image Classification
Image Classification
Meme Classification
Multilingual Word Embeddings
Object
Out-of-Distribution Generalization
Self-Supervised Image Classification
Self-Supervised Learning
Semi-Supervised Image Classification
Traffic Sign Recognition
Word Embeddings
Datasets
CIFAR-10
ImageNet
CIFAR-100
MNIST
KITTI
SVHN
UCF101
Kinetics
Oxford 102 Flower
STL-10
ImageNet-1K
Stanford Cars
DTD
Food-101
Caltech-101
CLEVR
Places205
iNaturalist
EuroSAT
FGVC-Aircraft
ImageNet-R
GTSRB
ImageNet-A
ImageNet-Sketch
Hateful Memes
ObjectNet
RESISC45
Kinetics-700
Oxford-IIIT Pet Dataset
Oxford-IIIT Pets
SUN397
ImageNet-W
Casual Conversations
Results from the Paper
Ranked #1 on
Copy Detection
on Copydays strong subset
(using extra training data)
