SGPT: GPT Sentence Embeddings for Semantic Search

17 Feb 2022  路  Niklas Muennighoff

Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Argument Retrieval ArguAna (BEIR) SGPT-CE-6.1B nDCG@10 0.286 # 5
Argument Retrieval ArguAna (BEIR) SGPT-BE-5.8B nDCG@10 0.514 # 2
Zero-shot Text Search BEIR SGPT-BE-5.8B (Muennighoff, 2022) Avg. Accuracy 55.0 # 1
Biomedical Information Retrieval BioASQ (BEIR) SGPT-CE-6.1B nDCG@10 0.547 # 2
Biomedical Information Retrieval BioASQ (BEIR) SGPT-CE-2.7B nDCG@10 0.546 # 3
Biomedical Information Retrieval BioASQ (BEIR) SGPT-BE-5.8B nDCG@10 0.413 # 6
Fact Checking CLIMATE-FEVER (BEIR) SGPT-CE-6.1B nDCG@10 0.161 # 4
Fact Checking CLIMATE-FEVER (BEIR) SGPT-BE-5.8B nDCG@10 0.305 # 1
Information Retrieval CQADupStack SGPT-BE-5.8B mAP@100 0.160 # 1
Duplicate-Question Retrieval CQADupStack (BEIR) SGPT-BE-5.8B nDCG@10 0.381 # 3
Duplicate-Question Retrieval CQADupStack (BEIR) SGPT-CE-6.1B nDCG@10 0.420 # 1
Entity Retrieval DBpedia (BEIR) SGPT-CE-6.1B nDCG@10 0.370 # 4
Entity Retrieval DBpedia (BEIR) SGPT-BE-5.8B nDCG@10 0.399 # 2
Fact Checking FEVER (BEIR) SGPT-CE-6.1B nDCG@10 0.725 # 4
Fact Checking FEVER (BEIR) SGPT-BE-5.8B nDCG@10 0.783 # 3
Question Answering FiQA-2018 (BEIR) SGPT-BE-5.8B nDCG@10 0.372 # 3
Question Answering FiQA-2018 (BEIR) SGPT-CE-6.1B nDCG@10 0.401 # 2
Question Answering HotpotQA (BEIR) SGPT-BE-5.8B nDCG@10 0.593 # 4
Question Answering HotpotQA (BEIR) SGPT-CE-6.1B nDCG@10 0.699 # 3
Passage Retrieval MSMARCO (BEIR) SGPT-CE-6.1B nDCG@10 0.290 # 9
Passage Retrieval MSMARCO (BEIR) SGPT-BE-5.8B nDCG@10 0.399 # 4
Passage Retrieval MSMARCO (BEIR) SGPT-CE-2.7B nDCG@10 0.278 # 10
Biomedical Information Retrieval NFCorpus (BEIR) SGPT-CE-2.7B nDCG@10 0.333 # 6
Biomedical Information Retrieval NFCorpus (BEIR) OpenAI Search-Davinci nDCG@10 0.358 # 3
Biomedical Information Retrieval NFCorpus (BEIR) SGPT-BE-5.8B nDCG@10 0.362 # 2
Biomedical Information Retrieval NFCorpus (BEIR) SGPT-CE-6.1B nDCG@10 0.347 # 5
Question Answering NQ (BEIR) SGPT-BE-5.8B nDCG@10 0.524 # 4
Question Answering NQ (BEIR) SGPT-CE-6.1B nDCG@10 0.401 # 6
Duplicate-Question Retrieval Quora (BEIR) SGPT-CE -6.1B nDCG@10 0.794 # 4
Duplicate-Question Retrieval Quora (BEIR) SGPT-BE-5.8B nDCG@10 0.846 # 1
Citation Prediction SciDocs (BEIR) SGPT-BE-5.8B nDCG@10 0.196 # 2
Citation Prediction SciDocs (BEIR) SGPT-CE-2.7B nDCG@10 0.186 # 4
Citation Prediction SciDocs (BEIR) SGPT-CE-6.1B nDCG@10 0.196 # 2
Fact Checking SciFact (BEIR) SGPT-CE-6.1B nDCG@10 0.682 # 4
Fact Checking SciFact (BEIR) SGPT-BE-5.8B nDCG@10 0.747 # 2
Tweet Retrieval Signal-1M (RT) (BEIR) SGPT-BE-5.8B nDCG@10 0.267 # 4
Tweet Retrieval Signal-1M (RT) (BEIR) SGPT-CE-6.1B nDCG@10 0.323 # 3
Argument Retrieval T贸uche-2020 (BEIR) SGPT-CE-6.1B nDCG@10 0.234 # 4
Argument Retrieval T贸uche-2020 (BEIR) SGPT-BE-5.8B nDCG@10 0.254 # 3
Biomedical Information Retrieval TREC-COVID (BEIR) SGPT-CE-6.1B nDCG@10 0.791 # 3
Biomedical Information Retrieval TREC-COVID (BEIR) SGPT-CE-2.7B nDCG@10 0.762 # 4
Biomedical Information Retrieval TREC-COVID (BEIR) SGPT-BE-5.8B nDCG@10 0.873 # 1
News Retrieval TREC-NEWS (BEIR) SGPT-CE-6.1B nDCG@10 0.466 # 3
News Retrieval TREC-NEWS (BEIR) SGPT-BE-5.8B nDCG@10 0.481 # 1
News Retrieval TREC-NEWS (BEIR) SGPT-CE-2.7B nDCG@10 0.434 # 4

Methods