Self-Supervised Learning

486 papers with code • 1 benchmarks • 21 datasets

Self-Supervised Learning is proposed for utilizing unlabeled data with the success of supervised learning. Producing a dataset with good labels is expensive, while unlabeled data is being generated all the time. The motivation of Self-Supervised Learning is to make use of the large amount of unlabeled data. The main idea of Self-Supervised Learning is to generate the labels from unlabeled data, according to the structure or characteristics of the data itself, and then train on this unsupervised data in a supervised manner. Self-Supervised Learning is wildly used in representation learning to make a model learn the latent features of the data. This technique is often employed in computer vision, video processing and robot control.

Source: Self-supervised Point Set Local Descriptors for Point Cloud Registration

Image source: LeCun

Greatest papers with code

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

tensorflow/models 22 Apr 2021

We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval.

 Ranked #1 on Action Classification on Moments in Time (using extra training data)

Action Classification Action Recognition +7

Spatiotemporal Contrastive Video Representation Learning

tensorflow/models CVPR 2021

Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.

Contrastive Learning Data Augmentation +4

Time-Contrastive Networks: Self-Supervised Learning from Video

tensorflow/models 23 Apr 2017

While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human.

Metric Learning Self-Supervised Learning +1

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

tensorflow/models ICLR 2020

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.

Common Sense Reasoning Linguistic Acceptability +4

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

huggingface/transformers 23 Oct 2020

We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT.

 Ranked #1 on Text Summarization on OrangeSum (using extra training data)

Natural Language Understanding Self-Supervised Learning +2

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

huggingface/transformers NeurIPS 2020

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler.

 Ranked #1 on Speech Recognition on TIMIT (using extra training data)

Quantization Self-Supervised Learning +1

Supervised Contrastive Learning

google-research/google-research NeurIPS 2020

Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models.

Contrastive Learning Data Augmentation +3

TabNet: Attentive Interpretable Tabular Learning

google-research/google-research 20 Aug 2019

We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet.

Decision Making Feature Selection +3

Temporal Cycle-Consistency Learning

google-research/google-research CVPR 2019

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos.

Anomaly Detection Representation Learning +2