SpanBERT: Improving Pre-training by Representing and Predicting Spans

24 Jul 2019Mandar JoshiDanqi ChenYinhan LiuDaniel S. WeldLuke ZettlemoyerOmer Levy

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it... (read more)

PDF Abstract

Evaluation Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK COMPARE
Linguistic Acceptability CoLA SpanBERT Accuracy 64.3% # 7
Question Answering HotpotQA SpanBERT Joint F1 83.0 # 1
Semantic Textual Similarity MRPC SpanBERT Accuracy 90.9% # 4
Question Answering NaturalQA SpanBERT F1 82.5 # 1
Question Answering NewsQA SpanBERT F1 73.6 # 1
Coreference Resolution OntoNotes SpanBERT-large F1 79.6 # 1
Coreference Resolution OntoNotes SpanBERT-base F1 77.7 # 2
Natural Language Inference QNLI SpanBERT Accuracy 94.3% # 9
Question Answering Quora Question Pairs SpanBERT Accuracy 89.5% # 9
Natural Language Inference RTE SpanBERT Accuracy 79,0% # 13
Open-Domain Question Answering SearchQA SpanBERT F1 84.8 # 1
Question Answering SQuAD1.1 SpanBERT (single model) EM 88.8 # 8
Question Answering SQuAD1.1 SpanBERT (single model) F1 94.6 # 5
Question Answering SQuAD2.0 SpanBERT EM 85.7 # 33
Question Answering SQuAD2.0 SpanBERT F1 88.7 # 29
Question Answering SQuAD2.0 dev SpanBERT F1 86.8 # 5
Sentiment Analysis SST-2 Binary classification SpanBERT Accuracy 94.8 # 11
Semantic Textual Similarity STS Benchmark SPANBert Pearson Correlation 0.899 # 7
Relation Extraction TACRED SpanBERT F1 70.8 # 2