Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

IJCNLP 2019  ·  Nils Reimers, Iryna Gurevych ·

BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT... The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods. read more

PDF Abstract IJCNLP 2019 PDF IJCNLP 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semantic Textual Similarity SICK SBERT-NLI-base Spearman Correlation 0.7291 # 6
Semantic Textual Similarity SICK SBERT-NLI-large Spearman Correlation 0.7375 # 5
Semantic Textual Similarity SICK SRoBERTa-NLI-large Spearman Correlation 0.7429 # 3
Semantic Textual Similarity SICK SRoBERTa-NLI-base Spearman Correlation 0.7446 # 2
Semantic Textual Similarity STS12 SRoBERTa-NLI-large Spearman Correlation 0.7453 # 2
Semantic Textual Similarity STS13 SBERT-NLI-large Spearman Correlation 0.7846 # 5
Semantic Textual Similarity STS14 SBERT-NLI-large Spearman Correlation 0.749 # 2
Semantic Textual Similarity STS15 SRoBERTa-NLI-large Spearman Correlation 0.8185 # 2
Semantic Textual Similarity STS16 SRoBERTa-NLI-large Spearman Correlation 0.7682 # 5
Semantic Textual Similarity STS Benchmark SRoBERTa-NLI-STSb-large Spearman Correlation 0.8615 # 10
Semantic Textual Similarity STS Benchmark SBERT-STSb-large Spearman Correlation 0.8445 # 15
Semantic Textual Similarity STS Benchmark SBERT-NLI-large Spearman Correlation 0.79 # 17
Semantic Textual Similarity STS Benchmark SBERT-NLI-base Spearman Correlation 0.7703 # 21
Semantic Textual Similarity STS Benchmark SRoBERTa-NLI-base Spearman Correlation 0.7777 # 20
Semantic Textual Similarity STS Benchmark SBERT-STSb-base Spearman Correlation 0.8535 # 12
Semantic Textual Similarity STS Benchmark SBERT-NLI-STSb-large Spearman Correlation 0.861 # 11
Semantic Textual Similarity STS Benchmark SRoBERTa-NLI-STSb-base Spearman Correlation 0.8479 # 14