SimCSE: Simple Contrastive Learning of Sentence Embeddings

EMNLP 2021  ·  Tianyu Gao, Xingcheng Yao, Danqi Chen ·

This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework, by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show -- both theoretically and empirically -- that contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.

PDF Abstract EMNLP 2021 PDF EMNLP 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Linear-Probe Classification SentEval SimCSE-supervised Accuracy 90.23 # 3
Linear-Probe Classification SentEval SimCSE-unsupervised Accuracy 87.6 # 6
Semantic Textual Similarity SICK SimCSE-RoBERTalarge Spearman Correlation 0.8195 # 2
Semantic Textual Similarity STS12 SimCSE-RoBERTa-base Spearman Correlation 0.7016 # 10
Semantic Textual Similarity STS12 SimCSE-RoBERTa-large Spearman Correlation 0.7746 # 4
Semantic Textual Similarity STS13 SimCSE-BERT-base Spearman Correlation 0.8241 # 10
Semantic Textual Similarity STS13 SimCSE-RoBERTa-base Spearman Correlation 0.8136 # 12
Semantic Textual Similarity STS13 SimCSE-RoBERTa-large Spearman Correlation 0.8727 # 5
Semantic Textual Similarity STS14 SimCSE-RoBERTalarge Spearman Correlation 0.8236 # 2
Semantic Textual Similarity STS15 SimCSE-RoBERTalarge Spearman Correlation 0.8666 # 4
Semantic Textual Similarity STS16 SimCSE-RoBERTalarge Spearman Correlation 0.8393 # 4
Semantic Textual Similarity STS Benchmark SimCSE-RoBERTalarge Spearman Correlation 0.867 # 10