Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure... To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT. read more

PDF Abstract NAACL 2021 PDF NAACL 2021 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Entity Linking KILT: AIDA-YAGO2 T5-base KILT-AC 74.05 # 5
R-Prec 74.05 # 5
Recall@5 74.05 # 6
Accuracy 74.05 # 6
Open-Domain Question Answering KILT: ELI5 T5-base KILT-RL 0.0 # 6
R-Prec 0.0 # 8
Recall@5 0.0 # 8
ROUGE-L 19.08 # 4
F1 16.1 # 8
KILT-F1 0.0 # 6
Fact Verification KILT: FEVER T5-base KILT-AC 0.0 # 9
R-Prec 0.0 # 11
Recall@5 0.0 # 11
Accuracy 76.3 # 8
Fact Verification KILT: FEVER RAG KILT-AC 53.45 # 6
R-Prec 61.94 # 8
Recall@5 75.55 # 7
Accuracy 86.31 # 5
Open-Domain Question Answering KILT: HotpotQA T5-base KILT-EM 0.0 # 6
R-Prec 0.0 # 8
Recall@5 0.0 # 8
EM 12.64 # 7
F1 19.57 # 7
KILT-F1 0.0 # 6
Open-Domain Question Answering KILT: Natural Questions T5-base KILT-EM 0.0 # 6
R-Prec 0.0 # 8
Recall@5 0.0 # 8
EM 19.6 # 7
F1 27.73 # 7
KILT-F1 0.0 # 6
Slot Filling KILT: T-REx T5-base KILT-AC 0.0 # 10
R-Prec 0.0 # 11
Recall@5 0.0 # 11
Accuracy 43.56 # 11
F1 50.61 # 10
KILT-F1 0.0 # 10
Open-Domain Question Answering KILT: TriviaQA T5-base KILT-EM 0.0 # 6
R-Prec 0.0 # 8
Recall@5 0.0 # 8
EM 18.11 # 8
F1 27.83 # 8
KILT-F1 0.0 # 6
Open-Domain Dialog KILT: Wizard of Wikipedia T5-base KILT-RL 0.0 # 10
R-Prec 0.0 # 12
Recall@5 0.0 # 12
ROUGE-L 12.4 # 9
F1 13.53 # 9
KILT-F1 0.0 # 10
Entity Linking KILT: WNED-CWEB T5-base KILT-AC 49.29 # 3
R-Prec 49.29 # 3
Recall@5 49.29 # 4
Accuracy 49.29 # 3
Entity Linking KILT: WNED-WIKI T5-base KILT-AC 47.13 # 4
R-Prec 47.13 # 5
Recall@5 47.13 # 5
Accuracy 47.13 # 4
Slot Filling KILT: Zero Shot RE T5-base KILT-AC 0.0 # 10
R-Prec 0.0 # 12
Recall@5 0.0 # 12
Accuracy 9.02 # 12
F1 13.52 # 12
KILT-F1 0.0 # 11

Methods