Data-Efficient Contrastive Learning by Differentiable Hard Sample and Hard Positive Pair Generation

Contrastive learning (CL), a self-supervised learning approach, can effectively learn visual representations from unlabeled data. However, CL requires learning on vast quantities of diverse data to achieve good performance, without which the performance of CL will greatly degrade. To tackle this problem, we propose a framework with two approaches to improve the data efficiency of CL training by generating beneficial samples. The first approach generates hard samples for the main model. In the training process, hard samples are dynamically customized to the training state of the main model, rather than fixed throughout the whole training process. With the progressively growing knowledge of the main model, the generated samples also become harder to constantly encourage the main model to learn better representations. Besides, a pair of data generators are proposed to generate similar but distinct samples as positive pairs. The hardness of positive pair is progressively increased by decreasing the similarity between a positive pair. In this way, the main model learns to cluster hard positives by pulling the representations of similar yet distinct samples together, by which the representations of similar samples are well-clustered and better representations can be learned. Comprehensive experiments show superior accuracy of the proposed approaches over the state-of-the-art on multiple datasets. For example, about 5% and 6% improvements are achieved on CIFAR-10/100 and ImageNet-100/10 with limited training data, respectively. Besides, about 2x data efficiency is achieved when reaching a similar accuracy as other methods.