BEIR (Benchmarking IR)
Introduced by Thakur et al. in BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
BEIR (Benchmarking IR) is a heterogeneous benchmark containing different information retrieval (IR) tasks. Through BEIR, it is possible to systematically study the zero-shot generalization capabilities of multiple neural retrieval approaches.
The benchmark contains a total of 9 information retrieval tasks (Fact Checking, Citation Prediction, Duplicate Question Retrieval, Argument Retrieval, News Retrieval, Question Answering, Tweet Retrieval, Biomedical IR, Entity Retrieval) from 19 different datasets:
Benchmarks
| Trend | Task | Dataset Variant | Best Model | Paper | Code |
|---|---|---|---|---|---|
|
BEIR
|
SGPT-BE-5.8B
|
|||
|
MSMARCO (BEIR)
|
BM25+CE
|
|||
|
NFCorpus (BEIR)
|
monoT5-3B
|
|||
|
NQ (BEIR)
|
Blended RAG
|
|||
|
BioASQ (BEIR)
|
monoT5-3B
|
|||
|
SciDocs (BEIR)
|
monoT5-3B
|
|||
|
TREC-COVID (BEIR)
|
SGPT-BE-5.8B
|
|||
|
CQADupStack (BEIR)
|
SGPT-CE-6.1B
|
|||
|
ArguAna (BEIR)
|
GenQ
|
|||
|
TREC-NEWS (BEIR)
|
SGPT-BE-5.8B
|
|||
|
SciFact (BEIR)
|
monoT5-3B
|
|||
|
FEVER (BEIR)
|
monoT5-3B
|
|||
|
CLIMATE-FEVER (BEIR)
|
SGPT-BE-5.8B
|
|||
|
HotpotQA (BEIR)
|
monoT5-3B
|
|||
|
Signal-1M (RT) (BEIR)
|
monoT5-3B
|
|||
|
FiQA-2018 (BEIR)
|
monoT5-3B
|
|||
|
Tóuche-2020 (BEIR)
|
monoT5-3B
|
|||
|
Quora (BEIR)
|
SGPT-BE-5.8B
|
|||
|
DBpedia (BEIR)
|
monoT5-3B
|
Papers
| Paper | Code | Results | Date | Stars |
|---|
