Autoregressive Entity Retrieval

Entities are at the center of how we represent and aggregate knowledge. For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per Wikipedia article). The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. Current approaches can be understood as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions. This approach has several shortcomings: (i) context and entity affinity is mainly captured through a vector dot product, potentially missing fine-grained interactions; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time. In this work, we propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion. This mitigates the aforementioned technical issues since: (i) the autoregressive formulation directly captures relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the softmax loss is computed without subsampling negative data. We experiment with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their names. Code and pre-trained models at https://github.com/facebookresearch/GENRE.

PDF Abstract ICLR 2021 PDF ICLR 2021 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Entity Disambiguation ACE2004 GENRE Micro-F1 90.1 # 5
Entity Disambiguation AIDA-CoNLL GENRE In-KB Accuracy 93.3 # 10
Entity Linking AIDA-CoNLL De Cao et al. (2021a) Micro-F1 strong 83.7 # 6
Entity Disambiguation AQUAINT GENRE Micro-F1 89.9 # 4
Entity Linking Derczynski De Cao et al. (2021a) Micro-F1 54.1 # 1
Entity Linking KILT: AIDA-YAGO2 GENRE KILT-AC 89.85 # 1
R-Prec 89.85 # 2
Recall@5 94.76 # 2
Accuracy 89.85 # 1
Entity Linking KILT: WNED-CWEB GENRE KILT-AC 71.22 # 1
R-Prec 71.22 # 1
Recall@5 79.22 # 2
Accuracy 71.22 # 1
Entity Linking KILT: WNED-WIKI GENRE KILT-AC 87.44 # 1
R-Prec 87.44 # 2
Recall@5 94.91 # 2
Accuracy 87.44 # 1
Entity Linking MSNBC De Cao et al. (2021a) Micro-F1 73.7 # 2
Entity Disambiguation MSNBC GENRE Micro-F1 94.3 # 4
Entity Disambiguation WNED-CWEB GENRE Micro-F1 77.3 # 5
Entity Disambiguation WNED-WIKI GENRE Micro-F1 87.4 # 4

Methods