TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Argument Retrieval	ArguAna (BEIR)	SGPT-CE-6.1B	nDCG@10	0.286	# 5
Argument Retrieval	ArguAna (BEIR)	SGPT-BE-5.8B	nDCG@10	0.514	# 2
Zero-shot Text Search	BEIR	SGPT-BE-5.8B (Muennighoff, 2022)	Avg. Accuracy	55.0	# 1
Biomedical Information Retrieval	BioASQ (BEIR)	SGPT-CE-6.1B	nDCG@10	0.547	# 2
Biomedical Information Retrieval	BioASQ (BEIR)	SGPT-CE-2.7B	nDCG@10	0.546	# 3
Biomedical Information Retrieval	BioASQ (BEIR)	SGPT-BE-5.8B	nDCG@10	0.413	# 6
Fact Checking	CLIMATE-FEVER (BEIR)	SGPT-CE-6.1B	nDCG@10	0.161	# 4
Fact Checking	CLIMATE-FEVER (BEIR)	SGPT-BE-5.8B	nDCG@10	0.305	# 1
Information Retrieval	CQADupStack	SGPT-BE-5.8B	mAP@100	0.160	# 1
Duplicate-Question Retrieval	CQADupStack (BEIR)	SGPT-BE-5.8B	nDCG@10	0.381	# 3
Duplicate-Question Retrieval	CQADupStack (BEIR)	SGPT-CE-6.1B	nDCG@10	0.420	# 1
Entity Retrieval	DBpedia (BEIR)	SGPT-CE-6.1B	nDCG@10	0.370	# 4
Entity Retrieval	DBpedia (BEIR)	SGPT-BE-5.8B	nDCG@10	0.399	# 2
Fact Checking	FEVER (BEIR)	SGPT-CE-6.1B	nDCG@10	0.725	# 4
Fact Checking	FEVER (BEIR)	SGPT-BE-5.8B	nDCG@10	0.783	# 3
Question Answering	FiQA-2018 (BEIR)	SGPT-BE-5.8B	nDCG@10	0.372	# 3
Question Answering	FiQA-2018 (BEIR)	SGPT-CE-6.1B	nDCG@10	0.401	# 2
Question Answering	HotpotQA (BEIR)	SGPT-BE-5.8B	nDCG@10	0.593	# 4
Question Answering	HotpotQA (BEIR)	SGPT-CE-6.1B	nDCG@10	0.699	# 3
Passage Retrieval	MSMARCO (BEIR)	SGPT-CE-6.1B	nDCG@10	0.290	# 9
Passage Retrieval	MSMARCO (BEIR)	SGPT-BE-5.8B	nDCG@10	0.399	# 4
Passage Retrieval	MSMARCO (BEIR)	SGPT-CE-2.7B	nDCG@10	0.278	# 10
Biomedical Information Retrieval	NFCorpus (BEIR)	SGPT-CE-2.7B	nDCG@10	0.333	# 6
Biomedical Information Retrieval	NFCorpus (BEIR)	OpenAI Search-Davinci	nDCG@10	0.358	# 3
Biomedical Information Retrieval	NFCorpus (BEIR)	SGPT-BE-5.8B	nDCG@10	0.362	# 2
Biomedical Information Retrieval	NFCorpus (BEIR)	SGPT-CE-6.1B	nDCG@10	0.347	# 5
Question Answering	NQ (BEIR)	SGPT-BE-5.8B	nDCG@10	0.524	# 4
Question Answering	NQ (BEIR)	SGPT-CE-6.1B	nDCG@10	0.401	# 6
Duplicate-Question Retrieval	Quora (BEIR)	SGPT-CE -6.1B	nDCG@10	0.794	# 4
Duplicate-Question Retrieval	Quora (BEIR)	SGPT-BE-5.8B	nDCG@10	0.846	# 1
Citation Prediction	SciDocs (BEIR)	SGPT-BE-5.8B	nDCG@10	0.196	# 2
Citation Prediction	SciDocs (BEIR)	SGPT-CE-2.7B	nDCG@10	0.186	# 4
Citation Prediction	SciDocs (BEIR)	SGPT-CE-6.1B	nDCG@10	0.196	# 2
Fact Checking	SciFact (BEIR)	SGPT-CE-6.1B	nDCG@10	0.682	# 4
Fact Checking	SciFact (BEIR)	SGPT-BE-5.8B	nDCG@10	0.747	# 2
Tweet Retrieval	Signal-1M (RT) (BEIR)	SGPT-BE-5.8B	nDCG@10	0.267	# 4
Tweet Retrieval	Signal-1M (RT) (BEIR)	SGPT-CE-6.1B	nDCG@10	0.323	# 3
Argument Retrieval	Tóuche-2020 (BEIR)	SGPT-CE-6.1B	nDCG@10	0.234	# 4
Argument Retrieval	Tóuche-2020 (BEIR)	SGPT-BE-5.8B	nDCG@10	0.254	# 3
Biomedical Information Retrieval	TREC-COVID (BEIR)	SGPT-CE-6.1B	nDCG@10	0.791	# 3
Biomedical Information Retrieval	TREC-COVID (BEIR)	SGPT-CE-2.7B	nDCG@10	0.762	# 4
Biomedical Information Retrieval	TREC-COVID (BEIR)	SGPT-BE-5.8B	nDCG@10	0.873	# 1
News Retrieval	TREC-NEWS (BEIR)	SGPT-CE-6.1B	nDCG@10	0.466	# 3
News Retrieval	TREC-NEWS (BEIR)	SGPT-BE-5.8B	nDCG@10	0.481	# 1
News Retrieval	TREC-NEWS (BEIR)	SGPT-CE-2.7B	nDCG@10	0.434	# 4

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/zero-shot-text-search-on-beir)](https://paperswithcode.com/sota/zero-shot-text-search-on-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/fact-checking-on-climate-fever-beir)](https://paperswithcode.com/sota/fact-checking-on-climate-fever-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/information-retrieval-on-cqadupstack)](https://paperswithcode.com/sota/information-retrieval-on-cqadupstack?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/duplicate-question-retrieval-on-cqadupstack-1)](https://paperswithcode.com/sota/duplicate-question-retrieval-on-cqadupstack-1?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/duplicate-question-retrieval-on-quora-beir)](https://paperswithcode.com/sota/duplicate-question-retrieval-on-quora-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/biomedical-information-retrieval-on-trec-1)](https://paperswithcode.com/sota/biomedical-information-retrieval-on-trec-1?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/news-retrieval-on-trec-news-beir)](https://paperswithcode.com/sota/news-retrieval-on-trec-news-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/argument-retrieval-on-arguana-beir)](https://paperswithcode.com/sota/argument-retrieval-on-arguana-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/biomedical-information-retrieval-on-bioasq-1)](https://paperswithcode.com/sota/biomedical-information-retrieval-on-bioasq-1?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/entity-retrieval-on-dbpedia-beir)](https://paperswithcode.com/sota/entity-retrieval-on-dbpedia-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/question-answering-on-fiqa-2018-beir)](https://paperswithcode.com/sota/question-answering-on-fiqa-2018-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/biomedical-information-retrieval-on-nfcorpus-1)](https://paperswithcode.com/sota/biomedical-information-retrieval-on-nfcorpus-1?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/citation-prediction-on-scidocs-beir)](https://paperswithcode.com/sota/citation-prediction-on-scidocs-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/fact-checking-on-scifact-beir)](https://paperswithcode.com/sota/fact-checking-on-scifact-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/fact-checking-on-fever-beir)](https://paperswithcode.com/sota/fact-checking-on-fever-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/question-answering-on-hotpotqa-beir)](https://paperswithcode.com/sota/question-answering-on-hotpotqa-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/tweet-retrieval-on-signal-1m-rt-beir)](https://paperswithcode.com/sota/tweet-retrieval-on-signal-1m-rt-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/argument-retrieval-on-touche-2020-beir)](https://paperswithcode.com/sota/argument-retrieval-on-touche-2020-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/passage-retrieval-on-msmarco-beir)](https://paperswithcode.com/sota/passage-retrieval-on-msmarco-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/sgpt-gpt-sentence-embeddings-for-semantic/question-answering-on-nq-beir)](https://paperswithcode.com/sota/question-answering-on-nq-beir?p=sgpt-gpt-sentence-embeddings-for-semantic)`

SGPT: GPT Sentence Embeddings for Semantic Search

17 Feb 2022 · Niklas Muennighoff ·

Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.

PDF Abstract

Code

Add Remove Mark official

muennighoff/sgpt official

807

Tasks

Add Remove

Argument Retrieval

Biomedical Information Retrieval

Citation Prediction

Duplicate-Question Retrieval

Entity Retrieval

Fact Checking

Information Retrieval

News Retrieval

Passage Retrieval

Question Answering

Sentence

Sentence Embeddings

Tweet Retrieval

Zero-shot Text Search

Datasets

Natural Questions

MS MARCO

HotpotQA

BEIR TREC-COVID CQADupStack FIQA

Results from the Paper

Edit

Ranked #1 on Information Retrieval on CQADupStack

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Argument Retrieval	ArguAna (BEIR)	SGPT-CE-6.1B	nDCG@10	0.286	# 5	Compare
Argument Retrieval	ArguAna (BEIR)	SGPT-BE-5.8B	nDCG@10	0.514	# 2	Compare
Zero-shot Text Search	BEIR	SGPT-BE-5.8B (Muennighoff, 2022)	Avg. Accuracy	55.0	# 1	Compare
Biomedical Information Retrieval	BioASQ (BEIR)	SGPT-CE-6.1B	nDCG@10	0.547	# 2	Compare
Biomedical Information Retrieval	BioASQ (BEIR)	SGPT-CE-2.7B	nDCG@10	0.546	# 3	Compare
Biomedical Information Retrieval	BioASQ (BEIR)	SGPT-BE-5.8B	nDCG@10	0.413	# 6	Compare
Fact Checking	CLIMATE-FEVER (BEIR)	SGPT-CE-6.1B	nDCG@10	0.161	# 4	Compare
Fact Checking	CLIMATE-FEVER (BEIR)	SGPT-BE-5.8B	nDCG@10	0.305	# 1	Compare
Information Retrieval	CQADupStack	SGPT-BE-5.8B	mAP@100	0.160	# 1	Compare
Duplicate-Question Retrieval	CQADupStack (BEIR)	SGPT-BE-5.8B	nDCG@10	0.381	# 3	Compare
Duplicate-Question Retrieval	CQADupStack (BEIR)	SGPT-CE-6.1B	nDCG@10	0.420	# 1	Compare
Entity Retrieval	DBpedia (BEIR)	SGPT-CE-6.1B	nDCG@10	0.370	# 4	Compare
Entity Retrieval	DBpedia (BEIR)	SGPT-BE-5.8B	nDCG@10	0.399	# 2	Compare
Fact Checking	FEVER (BEIR)	SGPT-CE-6.1B	nDCG@10	0.725	# 4	Compare
Fact Checking	FEVER (BEIR)	SGPT-BE-5.8B	nDCG@10	0.783	# 3	Compare
Question Answering	FiQA-2018 (BEIR)	SGPT-BE-5.8B	nDCG@10	0.372	# 3	Compare
Question Answering	FiQA-2018 (BEIR)	SGPT-CE-6.1B	nDCG@10	0.401	# 2	Compare
Question Answering	HotpotQA (BEIR)	SGPT-BE-5.8B	nDCG@10	0.593	# 4	Compare
Question Answering	HotpotQA (BEIR)	SGPT-CE-6.1B	nDCG@10	0.699	# 3	Compare
Passage Retrieval	MSMARCO (BEIR)	SGPT-CE-6.1B	nDCG@10	0.290	# 9	Compare
Passage Retrieval	MSMARCO (BEIR)	SGPT-BE-5.8B	nDCG@10	0.399	# 4	Compare
Passage Retrieval	MSMARCO (BEIR)	SGPT-CE-2.7B	nDCG@10	0.278	# 10	Compare
Biomedical Information Retrieval	NFCorpus (BEIR)	SGPT-CE-2.7B	nDCG@10	0.333	# 6	Compare
Biomedical Information Retrieval	NFCorpus (BEIR)	OpenAI Search-Davinci	nDCG@10	0.358	# 3	Compare
Biomedical Information Retrieval	NFCorpus (BEIR)	SGPT-BE-5.8B	nDCG@10	0.362	# 2	Compare
Biomedical Information Retrieval	NFCorpus (BEIR)	SGPT-CE-6.1B	nDCG@10	0.347	# 5	Compare
Question Answering	NQ (BEIR)	SGPT-BE-5.8B	nDCG@10	0.524	# 4	Compare
Question Answering	NQ (BEIR)	SGPT-CE-6.1B	nDCG@10	0.401	# 6	Compare
Duplicate-Question Retrieval	Quora (BEIR)	SGPT-CE -6.1B	nDCG@10	0.794	# 4	Compare
Duplicate-Question Retrieval	Quora (BEIR)	SGPT-BE-5.8B	nDCG@10	0.846	# 1	Compare
Citation Prediction	SciDocs (BEIR)	SGPT-BE-5.8B	nDCG@10	0.196	# 2	Compare
Citation Prediction	SciDocs (BEIR)	SGPT-CE-2.7B	nDCG@10	0.186	# 4	Compare
Citation Prediction	SciDocs (BEIR)	SGPT-CE-6.1B	nDCG@10	0.196	# 2	Compare
Fact Checking	SciFact (BEIR)	SGPT-CE-6.1B	nDCG@10	0.682	# 4	Compare
Fact Checking	SciFact (BEIR)	SGPT-BE-5.8B	nDCG@10	0.747	# 2	Compare
Tweet Retrieval	Signal-1M (RT) (BEIR)	SGPT-BE-5.8B	nDCG@10	0.267	# 4	Compare
Tweet Retrieval	Signal-1M (RT) (BEIR)	SGPT-CE-6.1B	nDCG@10	0.323	# 3	Compare
Argument Retrieval	Tóuche-2020 (BEIR)	SGPT-CE-6.1B	nDCG@10	0.234	# 4	Compare
Argument Retrieval	Tóuche-2020 (BEIR)	SGPT-BE-5.8B	nDCG@10	0.254	# 3	Compare
Biomedical Information Retrieval	TREC-COVID (BEIR)	SGPT-CE-6.1B	nDCG@10	0.791	# 3	Compare
Biomedical Information Retrieval	TREC-COVID (BEIR)	SGPT-CE-2.7B	nDCG@10	0.762	# 4	Compare
Biomedical Information Retrieval	TREC-COVID (BEIR)	SGPT-BE-5.8B	nDCG@10	0.873	# 1	Compare
News Retrieval	TREC-NEWS (BEIR)	SGPT-CE-6.1B	nDCG@10	0.466	# 3	Compare
News Retrieval	TREC-NEWS (BEIR)	SGPT-BE-5.8B	nDCG@10	0.481	# 1	Compare
News Retrieval	TREC-NEWS (BEIR)	SGPT-CE-2.7B	nDCG@10	0.434	# 4	Compare

Methods

Add Remove

Adam • Attention Dropout • BERT • BPE • Cosine Annealing • Dense Connections • Discriminative Fine-Tuning • Dropout • GELU • GPT • Layer Normalization • Linear Layer • Linear Warmup With Cosine Annealing • Linear Warmup With Linear Decay • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Weight Decay • WordPiece

Edit Social Preview

SGPT: GPT Sentence Embeddings for Semantic Search

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove