DeepStruct: Pretraining of Language Models for Structure Prediction

We introduce a method for improving the structural understanding abilities of language models. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate.

PDF Abstract Findings (ACL) 2022 PDF Findings (ACL) 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Joint Entity and Relation Extraction ACE2005 Deepstruct zero-shot Entity F1 31.8 # 3
Relation F1 5.3 # 3
Event Extraction ACE2005 DeepStruct multi-task w/ finetune Trigger Id 73.5 # 1
Trigger Cl 69.8 # 2
Argument Id 59.4 # 2
Argument Cl 56.2 # 2
Joint Entity and Relation Extraction ACE2005 DeepStruct multi-task w/ finetune Entity F1 90.0 # 2
Relation F1 66.8 # 1
Named Entity Recognition (NER) ACE2005 Deepstruct zero-shot F1 28.1 # 2
Joint Entity and Relation Extraction ACE2005 DeepStruct multi-task Entity F1 90.2 # 1
Relation F1 58.9 # 2
Event Extraction ACE2005 DeepStruct multi-task Trigger Id 72.7 # 2
Trigger Cl 69.2 # 3
Argument Id 67.5 # 1
Argument Cl 63.9 # 1
Named Entity Recognition (NER) ACE2005 DeepStruct multi-task w/ finetune F1 86.9 # 1
Joint Entity and Relation Extraction ADE Corpus DeepStruct multi-task w/ finetune Entity F1 91.1 # 1
Relation F1 83.8 # 1
Joint Entity and Relation Extraction ADE Corpus DeepStruct multi-task Entity F1 90.5 # 2
Relation F1 83.6 # 2
Joint Entity and Relation Extraction ADE Corpus Deepstruct zero-shot Entity F1 60.7 # 3
Relation F1 10.6 # 3
Intent Detection ATIS DeepStruct multi-task w/ finetune F1 97.8 # 1
Intent Detection ATIS DeepStruct multi-task F1 97.3 # 2
Named Entity Recognition (NER) CoNLL03 Deepstruct zero-shot F1 44.4 # 3
Named Entity Recognition (NER) CoNLL03 DeepStruct multi-task F1 93.1 # 1
Named Entity Recognition (NER) CoNLL03 DeepStruct multi-task w/ finetune F1 93.0 # 2
Joint Entity and Relation Extraction CoNLL04 DeepStruct multi-task w/ finetune Relation F1 78.3 # 1
Entity F1 90.7 # 1
Joint Entity and Relation Extraction CoNLL04 Deepstruct zero-shot Relation F1 25.8 # 3
Entity F1 48.3 # 3
Joint Entity and Relation Extraction CoNLL04 DeepStruct multi-task Relation F1 72.8 # 2
Entity F1 88.4 # 2
Semantic Role Labeling CoNLL05 Brown DeepStruct multi-task w/ finetune F1 92.1 # 1
Semantic Role Labeling CoNLL05 Brown DeepStruct multi-task F1 92.0 # 2
Semantic Role Labeling CoNLL05 WSJ DeepStruct multi-task w/ finetune F1 95.2 # 2
Semantic Role Labeling CoNLL05 WSJ DeepStruct multi-task F1 95.5 # 1
Coreference Resolution CoNLL12 DeepStruct multi-task MUC 63.9 # 2
B3 57.7 # 2
CEAFϕ4 60.2 # 2
Average F1 60.6 # 2
Coreference Resolution CoNLL12 DeepStruct multi-task w/ finetune MUC 74.9 # 1
B3 71.3 # 1
CEAFϕ4 73.1 # 1
Average F1 73.1 # 1
Semantic Role Labeling CoNLL12 DeepStruct multi-task F1 97.2 # 1
Semantic Role Labeling CoNLL12 DeepStruct multi-task w/ finetune F1 96.0 # 2
Relation Classification FewRel DeepStruct multi-task F1 (5-way 1-shot) 93.6 # 2
F1 (5-way 5-shot 96.4 # 2
F1 (10-way 1-shot) 92.2 # 2
F1 (10-way 5-shot) 94.6 # 2
Relation Classification FewRel DeepStruct multi-task w/ finetune F1 (5-way 1-shot) 98.4 # 1
F1 (5-way 5-shot 100 # 1
F1 (10-way 1-shot) 97.8 # 1
F1 (10-way 5-shot) 99.8 # 1
Relation Classification FewRel Deepstruct zero-shot F1 (5-way 1-shot) 72.4 # 3
F1 (5-way 5-shot 70.8 # 3
F1 (10-way 1-shot) 67.6 # 3
F1 (10-way 5-shot) 66.4 # 3
Named Entity Recognition (NER) GENIA DeepStruct multi-task w/ finetune F1 80.8 # 1
Named Entity Recognition (NER) GENIA DeepStruct multi-task F1 80.2 # 3
Named Entity Recognition (NER) GENIA Deepstruct zero-shot F1 47.2 # 13
Factual probe Google-RE Deepstruct zero-shot P@1 97.9 # 1
Factual probe Google-RE DeepStruct multi-task P@1 90.3 # 2
Dialogue State Tracking MULTIWOZ 2.1 DeepStruct multi-task w/ finetune Joint Acc 54.2 # 1
Dialogue State Tracking MULTIWOZ 2.1 DeepStruct multi-task Joint Acc 53.5 # 2
Joint Entity and Relation Extraction NYT DeepStruct multi-task Entity F1 95.4 # 2
Relation F1 93.7 # 1
Joint Entity and Relation Extraction NYT DeepStruct multi-task w/ finetune Entity F1 95.9 # 1
Relation F1 93.3 # 2
Joint Entity and Relation Extraction NYT Deepstruct zero-shot Entity F1 60.5 # 3
Relation F1 28.6 # 3
Open Information Extraction NYT DeepStruct multi-task w/ finetune F1 45.0 # 3
Open Information Extraction NYT Deepstruct zero-shot F1 28.9 # 1
Open Information Extraction NYT DeepStruct multi-task F1 43.6 # 2
Open Information Extraction OIE2016 Deepstruct zero-shot F1 28.1 # 9
Open Information Extraction OIE2016 DeepStruct multi-task w/ finetune F1 71.3 # 2
Open Information Extraction OIE2016 Deepstruct multi-task F1 71.2 # 3
Named Entity Recognition (NER) OntoNotes DeepStruct multi-task w/ finetune F1 87.8 # 1
Named Entity Recognition (NER) OntoNotes Deepstruct zero-shot F1 2.5 # 3
Named Entity Recognition (NER) OntoNotes DeepStruct multi-task F1 87.6 # 2
Open Information Extraction Penn Treebank Deepstruct zero-shot F1 51 # 1
Open Information Extraction Penn Treebank DeepStruct multi-task F1 54.5 # 2
Open Information Extraction Penn Treebank DeepStruct multi-task w/ finetune F1 45,1 # 4
Intent Detection SNIPS DeepStruct multi-task w/ finetune F1 97.3 # 2
Intent Detection SNIPS DeepStruct multi-task F1 97.4 # 1
Relation Classification TACRED DeepStruct multi-task F1 74.9 # 5
Relation Extraction TACRED DeepStruct multi-task w/ finetune F1 76.8 # 1
Relation Classification TACRED Deepstruct zero-shot F1 36.1 # 1
Relation Classification TACRED DeepStruct multi-task w/ finetune F1 76.8 # 7
Factual probe T-REx DeepStruct multi-task P@1 71.0 # 2
Factual probe T-REx Deepstruct zero-shot P@1 85.0 # 1
Open Information Extraction Web DeepStruct multi-task F1 50.8 # 3
Open Information Extraction Web DeepStruct multi-task w/ finetune F1 49.1 # 2
Open Information Extraction Web Deepstruct zero-shot F1 43.8 # 1

Methods


No methods listed for this paper. Add relevant methods here