TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Joint Entity and Relation Extraction	ACE2005	Deepstruct zero-shot	Entity F1	31.8	# 3
Joint Entity and Relation Extraction	ACE2005	Deepstruct zero-shot	Relation F1	5.3	# 3
Named Entity Recognition (NER)	ACE2005	DeepStruct multi-task w/ finetune	F1	86.9	# 1
Event Extraction	ACE2005	DeepStruct multi-task	Trigger Id	72.7	# 2
Event Extraction	ACE2005	DeepStruct multi-task	Trigger Cl	69.2	# 3
Event Extraction	ACE2005	DeepStruct multi-task	Argument Id	67.5	# 1
Event Extraction	ACE2005	DeepStruct multi-task	Argument Cl	63.9	# 1
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task	Entity F1	90.2	# 1
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task	Relation F1	58.9	# 2
Named Entity Recognition (NER)	ACE2005	Deepstruct zero-shot	F1	28.1	# 2
Event Extraction	ACE2005	DeepStruct multi-task w/ finetune	Trigger Id	73.5	# 1
Event Extraction	ACE2005	DeepStruct multi-task w/ finetune	Trigger Cl	69.8	# 2
Event Extraction	ACE2005	DeepStruct multi-task w/ finetune	Argument Id	59.4	# 2
Event Extraction	ACE2005	DeepStruct multi-task w/ finetune	Argument Cl	56.2	# 2
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task w/ finetune	Entity F1	90.0	# 2
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task w/ finetune	Relation F1	66.8	# 1
Joint Entity and Relation Extraction	ADE Corpus	Deepstruct zero-shot	Entity F1	60.7	# 3
Joint Entity and Relation Extraction	ADE Corpus	Deepstruct zero-shot	Relation F1	10.6	# 3
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task	Entity F1	90.5	# 2
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task	Relation F1	83.6	# 2
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task w/ finetune	Entity F1	91.1	# 1
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task w/ finetune	Relation F1	83.8	# 1
Intent Detection	ATIS	DeepStruct multi-task w/ finetune	F1	97.8	# 1
Intent Detection	ATIS	DeepStruct multi-task	F1	97.3	# 2
Named Entity Recognition (NER)	CoNLL03	Deepstruct zero-shot	F1	44.4	# 4
Named Entity Recognition (NER)	CoNLL03	DeepStruct multi-task w/ finetune	F1	93.0	# 3
Named Entity Recognition (NER)	CoNLL03	DeepStruct multi-task	F1	93.1	# 2
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task	Relation F1	72.8	# 2
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task	Entity F1	88.4	# 2
Joint Entity and Relation Extraction	CoNLL04	Deepstruct zero-shot	Relation F1	25.8	# 3
Joint Entity and Relation Extraction	CoNLL04	Deepstruct zero-shot	Entity F1	48.3	# 3
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task w/ finetune	Relation F1	78.3	# 1
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task w/ finetune	Entity F1	90.7	# 1
Semantic Role Labeling	CoNLL05 Brown	DeepStruct multi-task	F1	92.0	# 2
Semantic Role Labeling	CoNLL05 Brown	DeepStruct multi-task w/ finetune	F1	92.1	# 1
Semantic Role Labeling	CoNLL05 WSJ	DeepStruct multi-task	F1	95.5	# 1
Semantic Role Labeling	CoNLL05 WSJ	DeepStruct multi-task w/ finetune	F1	95.2	# 2
Semantic Role Labeling	CoNLL12	DeepStruct multi-task	F1	97.2	# 1
Coreference Resolution	CoNLL12	DeepStruct multi-task	MUC	63.9	# 2
Coreference Resolution	CoNLL12	DeepStruct multi-task	B3	57.7	# 2
Coreference Resolution	CoNLL12	DeepStruct multi-task	CEAFϕ4	60.2	# 2
Coreference Resolution	CoNLL12	DeepStruct multi-task	Average F1	60.6	# 2
Semantic Role Labeling	CoNLL12	DeepStruct multi-task w/ finetune	F1	96.0	# 2
Coreference Resolution	CoNLL12	DeepStruct multi-task w/ finetune	MUC	74.9	# 1
Coreference Resolution	CoNLL12	DeepStruct multi-task w/ finetune	B3	71.3	# 1
Coreference Resolution	CoNLL12	DeepStruct multi-task w/ finetune	CEAFϕ4	73.1	# 1
Coreference Resolution	CoNLL12	DeepStruct multi-task w/ finetune	Average F1	73.1	# 1
Relation Classification	FewRel	Deepstruct zero-shot	F1 (5-way 1-shot)	72.4	# 3
Relation Classification	FewRel	Deepstruct zero-shot	F1 (5-way 5-shot	70.8	# 3
Relation Classification	FewRel	Deepstruct zero-shot	F1 (10-way 1-shot)	67.6	# 3
Relation Classification	FewRel	Deepstruct zero-shot	F1 (10-way 5-shot)	66.4	# 3
Relation Classification	FewRel	DeepStruct multi-task w/ finetune	F1 (5-way 1-shot)	98.4	# 1
Relation Classification	FewRel	DeepStruct multi-task w/ finetune	F1 (5-way 5-shot	100	# 1
Relation Classification	FewRel	DeepStruct multi-task w/ finetune	F1 (10-way 1-shot)	97.8	# 1
Relation Classification	FewRel	DeepStruct multi-task w/ finetune	F1 (10-way 5-shot)	99.8	# 1
Relation Classification	FewRel	DeepStruct multi-task	F1 (5-way 1-shot)	93.6	# 2
Relation Classification	FewRel	DeepStruct multi-task	F1 (5-way 5-shot	96.4	# 2
Relation Classification	FewRel	DeepStruct multi-task	F1 (10-way 1-shot)	92.2	# 2
Relation Classification	FewRel	DeepStruct multi-task	F1 (10-way 5-shot)	94.6	# 2
Named Entity Recognition (NER)	GENIA	Deepstruct zero-shot	F1	47.2	# 14
Named Entity Recognition (NER)	GENIA	DeepStruct multi-task w/ finetune	F1	80.8	# 1
Named Entity Recognition (NER)	GENIA	DeepStruct multi-task	F1	80.2	# 3
Factual probe	Google-RE	Deepstruct zero-shot	P@1	97.9	# 1
Factual probe	Google-RE	DeepStruct multi-task	P@1	90.3	# 2
Dialogue State Tracking	MULTIWOZ 2.1	DeepStruct multi-task w/ finetune	Joint Acc	54.2	# 1
Dialogue State Tracking	MULTIWOZ 2.1	DeepStruct multi-task	Joint Acc	53.5	# 2
Joint Entity and Relation Extraction	NYT	Deepstruct zero-shot	Entity F1	60.5	# 3
Joint Entity and Relation Extraction	NYT	Deepstruct zero-shot	Relation F1	28.6	# 3
Open Information Extraction	NYT	DeepStruct multi-task w/ finetune	F1	45.0	# 3
Open Information Extraction	NYT	DeepStruct multi-task	F1	43.6	# 2
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task	Entity F1	95.4	# 2
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task	Relation F1	93.7	# 1
Open Information Extraction	NYT	Deepstruct zero-shot	F1	28.9	# 1
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task w/ finetune	Entity F1	95.9	# 1
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task w/ finetune	Relation F1	93.3	# 2
Open Information Extraction	OIE2016	Deepstruct zero-shot	F1	28.1	# 12
Open Information Extraction	OIE2016	DeepStruct multi-task w/ finetune	F1	71.3	# 2
Open Information Extraction	OIE2016	Deepstruct multi-task	F1	71.2	# 3
Named Entity Recognition (NER)	OntoNotes	DeepStruct multi-task w/ finetune	F1	87.8	# 1
Named Entity Recognition (NER)	OntoNotes	DeepStruct multi-task	F1	87.6	# 2
Named Entity Recognition (NER)	OntoNotes	Deepstruct zero-shot	F1	2.5	# 3
Open Information Extraction	Penn Treebank	Deepstruct zero-shot	F1	51	# 1
Open Information Extraction	Penn Treebank	DeepStruct multi-task	F1	54.5	# 2
Open Information Extraction	Penn Treebank	DeepStruct multi-task w/ finetune	F1	45,1	# 4
Intent Detection	SNIPS	DeepStruct multi-task	F1	97.4	# 1
Intent Detection	SNIPS	DeepStruct multi-task w/ finetune	F1	97.3	# 2
Relation Extraction	TACRED	DeepStruct multi-task w/ finetune	F1	76.8	# 1
Relation Classification	TACRED	Deepstruct zero-shot	F1	36.1	# 1
Relation Classification	TACRED	DeepStruct multi-task	F1	74.9	# 15
Relation Classification	TACRED	DeepStruct multi-task w/ finetune	F1	76.8	# 17
Factual probe	T-REx	DeepStruct multi-task	P@1	71.0	# 2
Factual probe	T-REx	Deepstruct zero-shot	P@1	85.0	# 1
Open Information Extraction	Web	Deepstruct zero-shot	F1	43.8	# 1
Open Information Extraction	Web	DeepStruct multi-task	F1	50.8	# 3
Open Information Extraction	Web	DeepStruct multi-task w/ finetune	F1	49.1	# 2

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/named-entity-recognition-on-ace2005)](https://paperswithcode.com/sota/named-entity-recognition-on-ace2005?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/event-extraction-on-ace2005)](https://paperswithcode.com/sota/event-extraction-on-ace2005?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/joint-entity-and-relation-extraction-on-7)](https://paperswithcode.com/sota/joint-entity-and-relation-extraction-on-7?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/joint-entity-and-relation-extraction-on-ade-1)](https://paperswithcode.com/sota/joint-entity-and-relation-extraction-on-ade-1?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/intent-detection-on-atis)](https://paperswithcode.com/sota/intent-detection-on-atis?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/joint-entity-and-relation-extraction-on-2)](https://paperswithcode.com/sota/joint-entity-and-relation-extraction-on-2?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/semantic-role-labeling-on-conll05-brown)](https://paperswithcode.com/sota/semantic-role-labeling-on-conll05-brown?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/semantic-role-labeling-on-conll05-wsj)](https://paperswithcode.com/sota/semantic-role-labeling-on-conll05-wsj?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/semantic-role-labeling-on-conll12)](https://paperswithcode.com/sota/semantic-role-labeling-on-conll12?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/coreference-resolution-on-conll12)](https://paperswithcode.com/sota/coreference-resolution-on-conll12?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/relation-classification-on-fewrel-1)](https://paperswithcode.com/sota/relation-classification-on-fewrel-1?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/named-entity-recognition-on-genia)](https://paperswithcode.com/sota/named-entity-recognition-on-genia?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/factual-probe-on-google-re)](https://paperswithcode.com/sota/factual-probe-on-google-re?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/dialogue-state-tracking-on-multiwoz-2-1)](https://paperswithcode.com/sota/dialogue-state-tracking-on-multiwoz-2-1?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/joint-entity-and-relation-extraction-on-nyt)](https://paperswithcode.com/sota/joint-entity-and-relation-extraction-on-nyt?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/open-information-extraction-on-nyt)](https://paperswithcode.com/sota/open-information-extraction-on-nyt?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/named-entity-recognition-on-ontonotes)](https://paperswithcode.com/sota/named-entity-recognition-on-ontonotes?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/open-information-extraction-on-penn-treebank)](https://paperswithcode.com/sota/open-information-extraction-on-penn-treebank?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/intent-detection-on-snips)](https://paperswithcode.com/sota/intent-detection-on-snips?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/relation-extraction-on-tacred)](https://paperswithcode.com/sota/relation-extraction-on-tacred?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/relation-classification-on-tacred-1)](https://paperswithcode.com/sota/relation-classification-on-tacred-1?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/factual-probe-on-t-rex)](https://paperswithcode.com/sota/factual-probe-on-t-rex?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/open-information-extraction-on-web)](https://paperswithcode.com/sota/open-information-extraction-on-web?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/named-entity-recognition-on-conll03)](https://paperswithcode.com/sota/named-entity-recognition-on-conll03?p=deepstruct-pretraining-of-language-models-for-1)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deepstruct-pretraining-of-language-models-for-1/open-information-extraction-on-oie2016)](https://paperswithcode.com/sota/open-information-extraction-on-oie2016?p=deepstruct-pretraining-of-language-models-for-1)`

DeepStruct: Pretraining of Language Models for Structure Prediction

Findings (ACL) 2022 · Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, Dawn Song ·

We introduce a method for improving the structural understanding abilities of language models. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate.

PDF Abstract Findings (ACL) 2022 PDF Findings (ACL) 2022 Abstract

Code

Add Remove Mark official

cgraywang/deepstruct official

Tasks

Add Remove

coreference-resolution

Coreference Resolution

Dialogue State Tracking

Event Extraction

Factual probe

Intent Detection

Joint Entity and Relation Extraction

Language Modelling

named-entity-recognition

Named Entity Recognition

Named Entity Recognition (NER)

Open Information Extraction

Relation

Relation Classification

Relation Extraction

Semantic Role Labeling

Datasets

Penn Treebank CoNLL 2003

MultiWOZ

New York Times Annotated Corpus

ATIS

SNIPS OntoNotes 5.0

TACRED CoNLL

FewRel

GENIA T-REx

ACE 2005 CoNLL++ OIE2016 KELM CoNLL04 TekGen OPIEC

Results from the Paper

Edit

Ranked #1 on Open Information Extraction on Penn Treebank

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Joint Entity and Relation Extraction	ACE2005	Deepstruct zero-shot	Entity F1	31.8	# 3	Compare
Joint Entity and Relation Extraction	ACE2005	Deepstruct zero-shot	Relation F1	5.3	# 3	Compare
Named Entity Recognition (NER)	ACE2005	DeepStruct multi-task w/ finetune	F1	86.9	# 1	Compare
Event Extraction	ACE2005	DeepStruct multi-task	Trigger Id	72.7	# 2	Compare
			Trigger Cl	69.2	# 3	Compare
			Argument Id	67.5	# 1	Compare
			Argument Cl	63.9	# 1	Compare
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task	Entity F1	90.2	# 1	Compare
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task	Relation F1	58.9	# 2	Compare
Named Entity Recognition (NER)	ACE2005	Deepstruct zero-shot	F1	28.1	# 2	Compare
Event Extraction	ACE2005	DeepStruct multi-task w/ finetune	Trigger Id	73.5	# 1	Compare
			Trigger Cl	69.8	# 2	Compare
			Argument Id	59.4	# 2	Compare
			Argument Cl	56.2	# 2	Compare
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task w/ finetune	Entity F1	90.0	# 2	Compare
Joint Entity and Relation Extraction	ACE2005	DeepStruct multi-task w/ finetune	Relation F1	66.8	# 1	Compare
Joint Entity and Relation Extraction	ADE Corpus	Deepstruct zero-shot	Entity F1	60.7	# 3	Compare
Joint Entity and Relation Extraction	ADE Corpus	Deepstruct zero-shot	Relation F1	10.6	# 3	Compare
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task	Entity F1	90.5	# 2	Compare
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task	Relation F1	83.6	# 2	Compare
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task w/ finetune	Entity F1	91.1	# 1	Compare
Joint Entity and Relation Extraction	ADE Corpus	DeepStruct multi-task w/ finetune	Relation F1	83.8	# 1	Compare
Intent Detection	ATIS	DeepStruct multi-task w/ finetune	F1	97.8	# 1	Compare
Intent Detection	ATIS	DeepStruct multi-task	F1	97.3	# 2	Compare
Named Entity Recognition (NER)	CoNLL03	Deepstruct zero-shot	F1	44.4	# 4	Compare
Named Entity Recognition (NER)	CoNLL03	DeepStruct multi-task w/ finetune	F1	93.0	# 3	Compare
Named Entity Recognition (NER)	CoNLL03	DeepStruct multi-task	F1	93.1	# 2	Compare
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task	Relation F1	72.8	# 2	Compare
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task	Entity F1	88.4	# 2	Compare
Joint Entity and Relation Extraction	CoNLL04	Deepstruct zero-shot	Relation F1	25.8	# 3	Compare
Joint Entity and Relation Extraction	CoNLL04	Deepstruct zero-shot	Entity F1	48.3	# 3	Compare
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task w/ finetune	Relation F1	78.3	# 1	Compare
Joint Entity and Relation Extraction	CoNLL04	DeepStruct multi-task w/ finetune	Entity F1	90.7	# 1	Compare
Semantic Role Labeling	CoNLL05 Brown	DeepStruct multi-task	F1	92.0	# 2	Compare
Semantic Role Labeling	CoNLL05 Brown	DeepStruct multi-task w/ finetune	F1	92.1	# 1	Compare
Semantic Role Labeling	CoNLL05 WSJ	DeepStruct multi-task	F1	95.5	# 1	Compare
Semantic Role Labeling	CoNLL05 WSJ	DeepStruct multi-task w/ finetune	F1	95.2	# 2	Compare
Semantic Role Labeling	CoNLL12	DeepStruct multi-task	F1	97.2	# 1	Compare
Coreference Resolution	CoNLL12	DeepStruct multi-task	MUC	63.9	# 2	Compare
			B3	57.7	# 2	Compare
			CEAFϕ4	60.2	# 2	Compare
			Average F1	60.6	# 2	Compare
Semantic Role Labeling	CoNLL12	DeepStruct multi-task w/ finetune	F1	96.0	# 2	Compare
Coreference Resolution	CoNLL12	DeepStruct multi-task w/ finetune	MUC	74.9	# 1	Compare
			B3	71.3	# 1	Compare
			CEAFϕ4	73.1	# 1	Compare
			Average F1	73.1	# 1	Compare
Relation Classification	FewRel	Deepstruct zero-shot	F1 (5-way 1-shot)	72.4	# 3	Compare
			F1 (5-way 5-shot	70.8	# 3	Compare
			F1 (10-way 1-shot)	67.6	# 3	Compare
			F1 (10-way 5-shot)	66.4	# 3	Compare
Relation Classification	FewRel	DeepStruct multi-task w/ finetune	F1 (5-way 1-shot)	98.4	# 1	Compare
			F1 (5-way 5-shot	100	# 1	Compare
			F1 (10-way 1-shot)	97.8	# 1	Compare
			F1 (10-way 5-shot)	99.8	# 1	Compare
Relation Classification	FewRel	DeepStruct multi-task	F1 (5-way 1-shot)	93.6	# 2	Compare
			F1 (5-way 5-shot	96.4	# 2	Compare
			F1 (10-way 1-shot)	92.2	# 2	Compare
			F1 (10-way 5-shot)	94.6	# 2	Compare
Named Entity Recognition (NER)	GENIA	Deepstruct zero-shot	F1	47.2	# 14	Compare
Named Entity Recognition (NER)	GENIA	DeepStruct multi-task w/ finetune	F1	80.8	# 1	Compare
Named Entity Recognition (NER)	GENIA	DeepStruct multi-task	F1	80.2	# 3	Compare
Factual probe	Google-RE	Deepstruct zero-shot	P@1	97.9	# 1	Compare
Factual probe	Google-RE	DeepStruct multi-task	P@1	90.3	# 2	Compare
Dialogue State Tracking	MULTIWOZ 2.1	DeepStruct multi-task w/ finetune	Joint Acc	54.2	# 1	Compare
Dialogue State Tracking	MULTIWOZ 2.1	DeepStruct multi-task	Joint Acc	53.5	# 2	Compare
Joint Entity and Relation Extraction	NYT	Deepstruct zero-shot	Entity F1	60.5	# 3	Compare
Joint Entity and Relation Extraction	NYT	Deepstruct zero-shot	Relation F1	28.6	# 3	Compare
Open Information Extraction	NYT	DeepStruct multi-task w/ finetune	F1	45.0	# 3	Compare
Open Information Extraction	NYT	DeepStruct multi-task	F1	43.6	# 2	Compare
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task	Entity F1	95.4	# 2	Compare
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task	Relation F1	93.7	# 1	Compare
Open Information Extraction	NYT	Deepstruct zero-shot	F1	28.9	# 1	Compare
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task w/ finetune	Entity F1	95.9	# 1	Compare
Joint Entity and Relation Extraction	NYT	DeepStruct multi-task w/ finetune	Relation F1	93.3	# 2	Compare
Open Information Extraction	OIE2016	Deepstruct zero-shot	F1	28.1	# 12	Compare
Open Information Extraction	OIE2016	DeepStruct multi-task w/ finetune	F1	71.3	# 2	Compare
Open Information Extraction	OIE2016	Deepstruct multi-task	F1	71.2	# 3	Compare
Named Entity Recognition (NER)	OntoNotes	DeepStruct multi-task w/ finetune	F1	87.8	# 1	Compare
Named Entity Recognition (NER)	OntoNotes	DeepStruct multi-task	F1	87.6	# 2	Compare
Named Entity Recognition (NER)	OntoNotes	Deepstruct zero-shot	F1	2.5	# 3	Compare
Open Information Extraction	Penn Treebank	Deepstruct zero-shot	F1	51	# 1	Compare
Open Information Extraction	Penn Treebank	DeepStruct multi-task	F1	54.5	# 2	Compare
Open Information Extraction	Penn Treebank	DeepStruct multi-task w/ finetune	F1	45,1	# 4	Compare
Intent Detection	SNIPS	DeepStruct multi-task	F1	97.4	# 1	Compare
Intent Detection	SNIPS	DeepStruct multi-task w/ finetune	F1	97.3	# 2	Compare
Relation Extraction	TACRED	DeepStruct multi-task w/ finetune	F1	76.8	# 1	Compare
Relation Classification	TACRED	Deepstruct zero-shot	F1	36.1	# 1	Compare
Relation Classification	TACRED	DeepStruct multi-task	F1	74.9	# 15	Compare
Relation Classification	TACRED	DeepStruct multi-task w/ finetune	F1	76.8	# 17	Compare
Factual probe	T-REx	DeepStruct multi-task	P@1	71.0	# 2	Compare
Factual probe	T-REx	Deepstruct zero-shot	P@1	85.0	# 1	Compare
Open Information Extraction	Web	Deepstruct zero-shot	F1	43.8	# 1	Compare
Open Information Extraction	Web	DeepStruct multi-task	F1	50.8	# 3	Compare
Open Information Extraction	Web	DeepStruct multi-task w/ finetune	F1	49.1	# 2	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

DeepStruct: Pretraining of Language Models for Structure Prediction

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove