On the Memorization Properties of Contrastive Learning
Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we investigate the memorization properties of SimCLR, a widely used contrastive self-supervised learning approach, and compare them to the memorization of supervised learning and random labels training. We find that both training objects and augmentations may have different complexity in the sense of how SimCLR learns them. Moreover, we show that SimCLR is similar to random labels training in terms of the distribution of training objects complexity.
PDF Abstract
Datasets
CIFAR-10
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
1x1 Convolution •
Average Pooling •
Batch Normalization •
Bottleneck Residual Block •
ColorJitter •
Convolution •
Dense Connections •
Feedforward Network •
Global Average Pooling •
Kaiming Initialization •
Max Pooling •
NT-Xent •
Random Gaussian Blur •
Random Resized Crop •
ReLU •
Residual Block •
Residual Connection •
ResNet •
SimCLR
