TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Zero-shot Text Search	ArguAna	COCO-DR Base	nDCG@10	49.3	# 2
Zero-shot Text Search	ArguAna	COCO-DR Large	nDCG@10	51.5	# 1
Zero-shot Text Search	BEIR	COCO-DR Base (Yu et al., 2022)	Avg. Accuracy	52.1	# 7
Zero-shot Text Search	BEIR	COCO-DR Base (Yu et al., 2022)	Avg. nDCG@10	46.2	# 3
Zero-shot Text Search	BEIR	COCO-DR Large (Yu et al., 2022)	Avg. Accuracy	54.1	# 2
Zero-shot Text Search	BEIR	COCO-DR Large (Yu et al., 2022)	Avg. nDCG@10	48.4	# 1
Zero-shot Text Search	BioASQ	COCO-DR Base	nDCG@10	42.9	# 2
Zero-shot Text Search	BioASQ	COCO-DR Large	nDCG@10	44.9	# 1
Zero-shot Text Search	CLIMATE-FEVER	COCO-DR Base	nDCG@10	21.1	# 2
Zero-shot Text Search	CLIMATE-FEVER	COCO-DR Large	nDCG@10	24.7	# 1
Zero-shot Text Search	CQADupStack	COCO-DR Base	nDCG@10	37	# 2
Zero-shot Text Search	CQADupStack	COCO-DR Large	nDCG@10	39.3	# 1
Zero-shot Text Search	DBpedia	COCO-DR Large	nDCG@10	40.7	# 1
Zero-shot Text Search	DBpedia	COCO-DR Base	nDCG@10	39.1	# 2
Zero-shot Text Search	FEVER	COCO-DR Base	nDCG@10	75.1	# 2
Zero-shot Text Search	FEVER	COCO-DR Large	nDCG@10	79.3	# 1
Zero-shot Text Search	FiQA-2018	COCO-DR Base	nDCG@10	30.7	# 2
Zero-shot Text Search	FiQA-2018	COCO-DR Large	nDCG@10	32.9	# 1
Zero-shot Text Search	NFCorpus	COCO-DR Large	nDCG@10	35.4	# 2
Zero-shot Text Search	NFCorpus	COCO-DR Base	nDCG@10	35.5	# 1
Zero-shot Text Search	NQ	COCO-DR Large	nDCG@10	54.7	# 2
Zero-shot Text Search	NQ	COCO-DR Base	nDCG@10	50.5	# 3
Zero-shot Text Search	quora	COCO-DR Large	nDCG@10	87.2	# 1
Zero-shot Text Search	quora	COCO-DR Base	nDCG@10	86.7	# 2
Zero-shot Text Search	Robust04	COCO-DR Base	nDCG@10	44.3	# 2
Zero-shot Text Search	Robust04	COCO-DR Large	nDCG@10	48.2	# 1
Zero-shot Text Search	SciDocs	COCO-DR Base	nDCG@10	16	# 2
Zero-shot Text Search	SciDocs	COCO-DR Large	nDCG@10	17.8	# 1
Zero-shot Text Search	SciFact	COCO-DR Large	nDCG@10	72.2	# 1
Zero-shot Text Search	SciFact	COCO-DR Base	nDCG@10	70.9	# 2
Zero-shot Text Search	Signal-1M (RT)	COCO-DR Large	nDCG@10	28.5	# 1
Zero-shot Text Search	Signal-1M (RT)	COCO-DR Base	nDCG@10	27.1	# 2
Zero-shot Text Search	TREC-COVID	COCO-DR Base	nDCG@10	78.9	# 3
Zero-shot Text Search	TREC-COVID	COCO-DR Large	nDCG@10	80.4	# 2
Zero-shot Text Search	TREC-COVID	GTR XXL	nDCG@10	50.1	# 7
Zero-shot Text Search	TREC-COVID	GTR XL	nDCG@10	58.4	# 6
Zero-shot Text Search	TREC-COVID	GPL	nDCG@10	70	# 4
Zero-shot Text Search	TREC-COVID	GenQ	nDCG@10	61.9	# 5
Zero-shot Text Search	TREC-News	COCO-DR Base	nDCG@10	40.3	# 2
Zero-shot Text Search	TREC-News	COCO-DR Large	nDCG@10	43.2	# 1
Zero-shot Text Search	Webis-Touché-2020	COCO-DR Base	nDCG@10	23.8	# 2
Zero-shot Text Search	Webis-Touché-2020	COCO-DR Large	nDCG@10	26.3	# 1

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-arguana)](https://paperswithcode.com/sota/zero-shot-text-search-on-arguana?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-bioasq)](https://paperswithcode.com/sota/zero-shot-text-search-on-bioasq?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-climate-fever)](https://paperswithcode.com/sota/zero-shot-text-search-on-climate-fever?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-cqadupstack)](https://paperswithcode.com/sota/zero-shot-text-search-on-cqadupstack?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-dbpedia)](https://paperswithcode.com/sota/zero-shot-text-search-on-dbpedia?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-fever)](https://paperswithcode.com/sota/zero-shot-text-search-on-fever?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-fiqa-2018)](https://paperswithcode.com/sota/zero-shot-text-search-on-fiqa-2018?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-nfcorpus)](https://paperswithcode.com/sota/zero-shot-text-search-on-nfcorpus?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-quora)](https://paperswithcode.com/sota/zero-shot-text-search-on-quora?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-robust04)](https://paperswithcode.com/sota/zero-shot-text-search-on-robust04?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-scidocs)](https://paperswithcode.com/sota/zero-shot-text-search-on-scidocs?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-scifact)](https://paperswithcode.com/sota/zero-shot-text-search-on-scifact?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-signal-1m-rt)](https://paperswithcode.com/sota/zero-shot-text-search-on-signal-1m-rt?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-trec-news)](https://paperswithcode.com/sota/zero-shot-text-search-on-trec-news?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-webis-touche-2020)](https://paperswithcode.com/sota/zero-shot-text-search-on-webis-touche-2020?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-beir)](https://paperswithcode.com/sota/zero-shot-text-search-on-beir?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-nq)](https://paperswithcode.com/sota/zero-shot-text-search-on-nq?p=coco-dr-combating-distribution-shifts-in-zero)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/coco-dr-combating-distribution-shifts-in-zero/zero-shot-text-search-on-trec-covid)](https://paperswithcode.com/sota/zero-shot-text-search-on-trec-covid?p=coco-dr-combating-distribution-shifts-in-zero)`

COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

27 Oct 2022 · Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, Arnold Overwijk ·

We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact of document differences, COCO-DR continues pretraining the language model on the target corpora to adapt the model to target distributions via COtinuous COtrastive learning. To prepare for unseen target queries, COCO-DR leverages implicit Distributionally Robust Optimization (iDRO) to reweight samples from different source query clusters for improving model robustness over rare queries during fine-tuning. COCO-DR achieves superior average performance on BEIR, the zero-shot retrieval benchmark. At BERT Base scale, COCO-DR Base outperforms other ZeroDR models with 60x larger size. At BERT Large scale, COCO-DR Large outperforms the giant GPT-3 embedding model which has 500x more parameters. Our analysis show the correlation between COCO-DR's effectiveness in combating distribution shifts and improving zero-shot accuracy. Our code and model can be found at \url{https://github.com/OpenMatch/COCO-DR}.

PDF Abstract

Code

Add Remove Mark official

openmatch/coco-dr official

Tasks

Add Remove

Language Modelling

Retrieval

Text Retrieval

Zero-shot Text Search

Datasets

Natural Questions

MS MARCO

HotpotQA

DBpedia

FEVER

BEIR

BioASQ SciFact TREC-COVID

SciDocs CLIMATE-FEVER Robust04 CQADupStack NFCorpus Webis-Touché-2020 TREC-News Signal-1M

Results from the Paper

Edit

Ranked #1 on Zero-shot Text Search on CQADupStack

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Zero-shot Text Search	ArguAna	COCO-DR Base	nDCG@10	49.3	# 2	Compare
Zero-shot Text Search	ArguAna	COCO-DR Large	nDCG@10	51.5	# 1	Compare
Zero-shot Text Search	BEIR	COCO-DR Base (Yu et al., 2022)	Avg. Accuracy	52.1	# 7	Compare
Zero-shot Text Search	BEIR	COCO-DR Base (Yu et al., 2022)	Avg. nDCG@10	46.2	# 3	Compare
Zero-shot Text Search	BEIR	COCO-DR Large (Yu et al., 2022)	Avg. Accuracy	54.1	# 2	Compare
Zero-shot Text Search	BEIR	COCO-DR Large (Yu et al., 2022)	Avg. nDCG@10	48.4	# 1	Compare
Zero-shot Text Search	BioASQ	COCO-DR Base	nDCG@10	42.9	# 2	Compare
Zero-shot Text Search	BioASQ	COCO-DR Large	nDCG@10	44.9	# 1	Compare
Zero-shot Text Search	CLIMATE-FEVER	COCO-DR Base	nDCG@10	21.1	# 2	Compare
Zero-shot Text Search	CLIMATE-FEVER	COCO-DR Large	nDCG@10	24.7	# 1	Compare
Zero-shot Text Search	CQADupStack	COCO-DR Base	nDCG@10	37	# 2	Compare
Zero-shot Text Search	CQADupStack	COCO-DR Large	nDCG@10	39.3	# 1	Compare
Zero-shot Text Search	DBpedia	COCO-DR Large	nDCG@10	40.7	# 1	Compare
Zero-shot Text Search	DBpedia	COCO-DR Base	nDCG@10	39.1	# 2	Compare
Zero-shot Text Search	FEVER	COCO-DR Base	nDCG@10	75.1	# 2	Compare
Zero-shot Text Search	FEVER	COCO-DR Large	nDCG@10	79.3	# 1	Compare
Zero-shot Text Search	FiQA-2018	COCO-DR Base	nDCG@10	30.7	# 2	Compare
Zero-shot Text Search	FiQA-2018	COCO-DR Large	nDCG@10	32.9	# 1	Compare
Zero-shot Text Search	NFCorpus	COCO-DR Large	nDCG@10	35.4	# 2	Compare
Zero-shot Text Search	NFCorpus	COCO-DR Base	nDCG@10	35.5	# 1	Compare
Zero-shot Text Search	NQ	COCO-DR Large	nDCG@10	54.7	# 2	Compare
Zero-shot Text Search	NQ	COCO-DR Base	nDCG@10	50.5	# 3	Compare
Zero-shot Text Search	quora	COCO-DR Large	nDCG@10	87.2	# 1	Compare
Zero-shot Text Search	quora	COCO-DR Base	nDCG@10	86.7	# 2	Compare
Zero-shot Text Search	Robust04	COCO-DR Base	nDCG@10	44.3	# 2	Compare
Zero-shot Text Search	Robust04	COCO-DR Large	nDCG@10	48.2	# 1	Compare
Zero-shot Text Search	SciDocs	COCO-DR Base	nDCG@10	16	# 2	Compare
Zero-shot Text Search	SciDocs	COCO-DR Large	nDCG@10	17.8	# 1	Compare
Zero-shot Text Search	SciFact	COCO-DR Large	nDCG@10	72.2	# 1	Compare
Zero-shot Text Search	SciFact	COCO-DR Base	nDCG@10	70.9	# 2	Compare
Zero-shot Text Search	Signal-1M (RT)	COCO-DR Large	nDCG@10	28.5	# 1	Compare
Zero-shot Text Search	Signal-1M (RT)	COCO-DR Base	nDCG@10	27.1	# 2	Compare
Zero-shot Text Search	TREC-COVID	COCO-DR Base	nDCG@10	78.9	# 3	Compare
Zero-shot Text Search	TREC-COVID	COCO-DR Large	nDCG@10	80.4	# 2	Compare
Zero-shot Text Search	TREC-COVID	GTR XXL	nDCG@10	50.1	# 7	Compare
Zero-shot Text Search	TREC-COVID	GTR XL	nDCG@10	58.4	# 6	Compare
Zero-shot Text Search	TREC-COVID	GPL	nDCG@10	70	# 4	Compare
Zero-shot Text Search	TREC-COVID	GenQ	nDCG@10	61.9	# 5	Compare
Zero-shot Text Search	TREC-News	COCO-DR Base	nDCG@10	40.3	# 2	Compare
Zero-shot Text Search	TREC-News	COCO-DR Large	nDCG@10	43.2	# 1	Compare
Zero-shot Text Search	Webis-Touché-2020	COCO-DR Base	nDCG@10	23.8	# 2	Compare
Zero-shot Text Search	Webis-Touché-2020	COCO-DR Large	nDCG@10	26.3	# 1	Compare

Methods

Add Remove

Adam • Attention Dropout • BASE • BERT • BPE • Cosine Annealing • Dense Connections • Dropout • Fixed Factorized Attention • GELU • GPT-3 • Layer Normalization • Linear Layer • Linear Warmup With Cosine Annealing • Linear Warmup With Linear Decay • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Strided Attention • Weight Decay • WordPiece

Edit Social Preview

COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove