Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Automatically inducing high quality knowledge graphs from a given collection of documents still remains a challenging problem in AI. One way to make headway for this problem is through advancements in a related task known as slot filling... In this task, given an entity query in form of [Entity, Slot, ? ], a system is asked to fill the slot by generating or extracting the missing value exploiting evidence extracted from relevant passage(s) in the given document collection. The recent works in the field try to solve this task in an end-to-end fashion using retrieval-based language models. In this paper, we present a novel approach to zero-shot slot filling that extends dense passage retrieval with hard negatives and robust training procedures for retrieval augmented generation models. Our model reports large improvements on both T-REx and zsRE slot filling datasets, improving both passage retrieval and slot value generation, and ranking at the top-1 position in the KILT leaderboard. Moreover, we demonstrate the robustness of our system showing its domain adaptation capability on a new variant of the TACRED dataset for slot filling, through a combination of zero/few-shot learning. We release the source code and pre-trained models. read more

PDF Abstract EMNLP 2021 PDF EMNLP 2021 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-shot Slot Filling T-REx DPRDNS+RAG R-Prec 74.34 # 1
R@5 82.89 # 1
Zero-shot Slot Filling T-REx DPRDNS R-Prec 42.62 # 5
R@5 55.09 # 5
Zero-shot Slot Filling T-REx DPRNQ R-Prec 19.5 # 6
R@5 29.8 # 6
Zero-shot Slot Filling T-REx DPRNQ+RAG R-Prec 53.04 # 3
R@5 65.54 # 3
Zero-shot Slot Filling T-REx DPRBM25 R-Prec 49.02 # 4
R@5 63.34 # 4
Zero-shot Slot Filling T-REx DPRBM25+RAG R-Prec 65.02 # 2
R@5 75.52 # 2
Zero-shot Slot Filling zsRE DPRNQ R-Prec 45.49 # 6
R@5 60.77 # 6
Zero-shot Slot Filling zsRE DPRDNS+RAG R-Prec 98.60 # 1
R@5 99.70 # 1
Zero-shot Slot Filling zsRE DPRBM25+RAG R-Prec 96.89 # 3
R@5 98.01 # 4
Zero-shot Slot Filling zsRE DPRDNS R-Prec 97.53 # 2
R@5 99.3 # 2
Zero-shot Slot Filling zsRE DPRNQ+RAG R-Prec 68.13 # 5
R@5 79.19 # 5
Zero-shot Slot Filling zsRE DPRBM25 R-Prec 94.55 # 4
R@5 98.17 # 3

Methods


No methods listed for this paper. Add relevant methods here