Self-Supervised Learning

402 papers with code • 1 benchmarks • 21 datasets

Self-Supervised Learning is proposed for utilizing unlabeled data with the success of supervised learning. Producing a dataset with good labels is expensive, while unlabeled data is being generated all the time. The motivation of Self-Supervised Learning is to make use of the large amount of unlabeled data. The main idea of Self-Supervised Learning is to generate the labels from unlabeled data, according to the structure or characteristics of the data itself, and then train on this unsupervised data in a supervised manner. Self-Supervised Learning is wildly used in representation learning to make a model learn the latent features of the data. This technique is often employed in computer vision, video processing and robot control.

Source: Self-supervised Point Set Local Descriptors for Point Cloud Registration

Image source: LeCun

Greatest papers with code

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

tensorflow/models 22 Apr 2021

We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks.

 Ranked #1 on Action Classification on Moments in Time (using extra training data)

Action Classification Action Recognition +6

Time-Contrastive Networks: Self-Supervised Learning from Video

tensorflow/models 23 Apr 2017

While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human.

Metric Learning Self-Supervised Learning +1

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

tensorflow/models ICLR 2020

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.

Common Sense Reasoning Linguistic Acceptability +4

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

huggingface/transformers 23 Oct 2020

We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT.

 Ranked #1 on Text Summarization on OrangeSum (using extra training data)

Natural Language Understanding Self-Supervised Learning +2

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

huggingface/transformers NeurIPS 2020

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler.

Quantization Self-Supervised Learning +1

Temporal Cycle-Consistency Learning

google-research/google-research CVPR 2019

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos.

Anomaly Detection Representation Learning +2

TabNet: Attentive Interpretable Tabular Learning

google-research/google-research 20 Aug 2019

We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet.

Decision Making Feature Selection +2

Supervised Contrastive Learning

google-research/google-research NeurIPS 2020

Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models.

Contrastive Learning Data Augmentation +3

A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

PyTorchLightning/pytorch-lightning 31 Aug 2020

Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task that selects and compares anchor, negative and positive (APN) features from an unlabeled dataset.

Data Augmentation Image Classification +1

Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training

pytorch/fairseq 2 Apr 2021

On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%.

Self-Supervised Learning