Self-supervised Pretraining of Visual Features in the Wild

Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT BENCHMARK
Self-Supervised Image Classification ImageNet SEER Large (RegNetY-256) Top 1 Accuracy 77.51% # 12
Number of Params 1300M # 1
Self-Supervised Image Classification ImageNet SEER Small (RegNetY-256) Top 1 Accuracy 75.95% # 20
Number of Params 693M # 3
Semi-Supervised Image Classification ImageNet - 10% labeled data SEER Large (RegNetY-256GF) Top 1 Accuracy 77.9% # 4
Semi-Supervised Image Classification ImageNet - 10% labeled data SEER Small (RegNetY-128GF) Top 1 Accuracy 76.7% # 6
Semi-Supervised Image Classification ImageNet - 1% labeled data SEER Large (RegNetY-256GF) Top 1 Accuracy 60.5% # 12
Semi-Supervised Image Classification ImageNet - 1% labeled data SEER Small (RegNetY-128GF) Top 1 Accuracy 57.5% # 15
Self-Supervised Image Classification ImageNet (finetuned) SEER Small (RegNetY-128GF) Number of Params 693M # 2
Top 1 Accuracy 83.8% # 2
Self-Supervised Image Classification ImageNet (finetuned) SEER Large (RegNetY-256GF) Number of Params 1300M # 1
Top 1 Accuracy 84.2% # 1

Methods used in the Paper


METHOD TYPE
Gradient Checkpointing
Stochastic Optimization
BYOL
Self-Supervised Learning
Residual Connection
Skip Connections
Bottleneck Residual Block
Skip Connection Blocks
Kaiming Initialization
Initialization
Residual Block
Skip Connection Blocks
ColorJitter
Image Data Augmentation
Max Pooling
Pooling Operations
ResNet
Convolutional Neural Networks
Sigmoid Activation
Activation Functions
Squeeze-and-Excitation Block
Image Model Blocks
Average Pooling
Pooling Operations
Grouped Convolution
Convolutions
Random Gaussian Blur
Image Data Augmentation
Dense Connections
Feedforward Networks
Feedforward Network
Feedforward Networks
1x1 Convolution
Convolutions
NT-Xent
Loss Functions
LARS
Large Batch Optimization
InfoNCE
Loss Functions
Convolution
Convolutions
Random Resized Crop
Image Data Augmentation
SimCLR
Self-Supervised Learning
Batch Normalization
Normalization
SwAV
Self-Supervised Learning
ReLU
Activation Functions
MoCo
Self-Supervised Learning
Global Average Pooling
Pooling Operations
RegNetY
Convolutional Neural Networks