GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

Large Language Models (LLMs) combined with instruction tuning have made significant progress when generalizing to unseen tasks. However, they have been less successful in Information Extraction (IE), lagging behind task-specific models. Typically, IE tasks are characterized by complex annotation guidelines that describe the task and give examples to humans. Previous attempts to leverage such information have failed, even with the largest models, as they are not able to follow the guidelines out of the box. In this paper, we propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. Comprehensive evaluation empirically demonstrates that GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous attempts at zero-shot information extraction. The ablation study shows that detailed guidelines are key for good results.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Named Entity Recognition (NER) ACE 2005 GoLLIE F1 89.6 # 2
Relation Extraction ACE 2005 GoLLIE RE Micro F1 70.1 # 3
Named Entity Recognition (NER) BC5CDR GoLLIE F1 88.4 # 13
Zero-shot Named Entity Recognition (NER) Broad Twitter Corpus GoLLIE Entity F1 51.4 # 1
Named Entity Recognition (NER) CoNLL 2003 (English) GoLLIE F1 93.1 # 26
Zero-shot Named Entity Recognition (NER) CrossNER GoLLIE AI 61.6 # 1
Literature 62.7 # 1
Music 68.4 # 1
Politics 60.2 # 1
Science 56.3 # 1
Zero-shot Named Entity Recognition (NER) HarveyNER GoLLIE Entity F1 41.3 # 1
Named Entity Recognition (NER) NCBI-disease GoLLIE F1 86.5 # 23
Event Argument Extraction WikiEvents GoLLIE F1 (Zero-Shot) 52.5 # 2
Zero-shot Named Entity Recognition (NER) WikiEvents GoLLIE Entity F1 81.3 # 1
Named Entity Recognition (NER) WNUT 2017 GoLLIE F1 54.3 # 9


No methods listed for this paper. Add relevant methods here