About

Self-Supervised Learning is proposed for utilizing unlabeled data with the success of supervised learning. Producing a dataset with good labels is expensive, while unlabeled data is being generated all the time. The motivation of Self-Supervised Learning is to make use of the large amount of unlabeled data. The main idea of Self-Supervised Learning is to generate the labels from unlabeled data, according to the structure or characteristics of the data itself, and then train on this unsupervised data in a supervised manner. Self-Supervised Learning is wildly used in representation learning to make a model learn the latent features of the data. This technique is often employed in computer vision, video processing and robot control.

Source: Self-supervised Point Set Local Descriptors for Point Cloud Registration

Image source: LeCun

Benchmarks

TREND DATASET BEST METHOD PAPER TITLE PAPER CODE COMPARE

Subtasks

Datasets

Greatest papers with code

Time-Contrastive Networks: Self-Supervised Learning from Video

23 Apr 2017tensorflow/models

While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human.

METRIC LEARNING SELF-SUPERVISED LEARNING VIDEO ALIGNMENT

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

23 Oct 2020huggingface/transformers

We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT.

 Ranked #1 on Text Summarization on OrangeSum (using extra training data)

NATURAL LANGUAGE UNDERSTANDING SELF-SUPERVISED LEARNING TEXT SUMMARIZATION TRANSFER LEARNING

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

NeurIPS 2020 huggingface/transformers

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler.

 Ranked #1 on Speech Recognition on TIMIT (using extra training data)

QUANTIZATION SELF-SUPERVISED LEARNING SPEECH RECOGNITION

Temporal Cycle-Consistency Learning

CVPR 2019 google-research/google-research

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos.

ANOMALY DETECTION REPRESENTATION LEARNING SELF-SUPERVISED LEARNING VIDEO ALIGNMENT

TabNet: Attentive Interpretable Tabular Learning

20 Aug 2019google-research/google-research

We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet.

DECISION MAKING FEATURE SELECTION SELF-SUPERVISED LEARNING UNSUPERVISED REPRESENTATION LEARNING

Supervised Contrastive Learning

NeurIPS 2020 google-research/google-research

Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models.

DATA AUGMENTATION IMAGE CLASSIFICATION REPRESENTATION LEARNING SELF-SUPERVISED LEARNING

A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

31 Aug 2020PyTorchLightning/pytorch-lightning

Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task that selects and compares anchor, negative and positive (APN) features from an unlabeled dataset.

DATA AUGMENTATION IMAGE CLASSIFICATION SELF-SUPERVISED LEARNING

vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

ICLR 2020 pytorch/fairseq

We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task.

Ranked #2 on Speech Recognition on TIMIT (using extra training data)

CLASSIFICATION SELF-SUPERVISED LEARNING SPEECH RECOGNITION