A Simple Framework for Contrastive Learning of Visual Representations

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank... (read more)

PDF Abstract ICML 2020 PDF
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Self-Supervised Image Classification ImageNet SimCLR (ResNet-50) Top 1 Accuracy 69.3% # 37
Top 5 Accuracy 89.0% # 18
Number of Params 24M # 27
Self-Supervised Image Classification ImageNet SimCLR (ResNet-50 4x) Top 1 Accuracy 76.5% # 17
Top 5 Accuracy 93.2% # 5
Number of Params 375M # 8
Self-Supervised Image Classification ImageNet SimCLR (ResNet-50 2x) Top 1 Accuracy 74.2% # 29
Top 5 Accuracy 92.0% # 9
Number of Params 94M # 16
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLR (ResNet-50 2×) Top 5 Accuracy 91.2% # 12
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLR (ResNet-50) Top 5 Accuracy 87.8% # 21
Semi-Supervised Image Classification ImageNet - 10% labeled data SimCLR (ResNet-50 4×) Top 5 Accuracy 92.6% # 5
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLR (ResNet-50 4×) Top 5 Accuracy 85.8% # 9
Top 1 Accuracy 63.0% # 10
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLR (ResNet-50 2×) Top 5 Accuracy 83.0% # 12
Top 1 Accuracy 58.5% # 13
Semi-Supervised Image Classification ImageNet - 1% labeled data SimCLR (ResNet-50) Top 5 Accuracy 75.5% # 21
Top 1 Accuracy 48.3% # 21

Methods used in the Paper


METHOD TYPE
Dense Connections
Feedforward Networks
NT-Xent
Loss Functions
Random Resized Crop
Image Data Augmentation
Random Gaussian Blur
Image Data Augmentation
ColorJitter
Image Data Augmentation
Feedforward Network
Feedforward Networks
Linear Warmup With Linear Decay
Learning Rate Schedules
Weight Decay
Regularization
LARS
Large Batch Optimization
SimCLR
Self-Supervised Learning
Average Pooling
Pooling Operations
Residual Connection
Skip Connections
ReLU
Activation Functions
1x1 Convolution
Convolutions
Batch Normalization
Normalization
Bottleneck Residual Block
Skip Connection Blocks
Global Average Pooling
Pooling Operations
Residual Block
Skip Connection Blocks
Kaiming Initialization
Initialization
Max Pooling
Pooling Operations
Convolution
Convolutions
ResNet
Convolutional Neural Networks