SpanBERT: Improving Pre-training by Representing and Predicting Spans

24 Jul 2019Mandar JoshiDanqi ChenYinhan LiuDaniel S. WeldLuke ZettlemoyerOmer Levy

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT LEADERBOARD
Linguistic Acceptability CoLA SpanBERT Accuracy 64.3% # 8
Question Answering HotpotQA SpanBERT Joint F1 83 # 1
Semantic Textual Similarity MRPC SpanBERT Accuracy 90.9% # 3
Natural Language Inference MultiNLI SpanBERT Matched 88.1 # 7
Question Answering NaturalQA SpanBERT F1 82.5 # 1
Question Answering NewsQA SpanBERT F1 73.6 # 1
Coreference Resolution OntoNotes SpanBERT F1 79.6 # 1
Natural Language Inference QNLI SpanBERT Accuracy 94.3% # 9
Paraphrase Identification Quora Question Pairs SpanBERT Accuracy 71.9 # 9
Natural Language Inference RTE SpanBERT Accuracy 79.0% # 10
Open-Domain Question Answering SearchQA SpanBERT F1 84.8 # 1
Question Answering SQuAD1.1 SpanBERT (single model) EM 88.8 # 8
F1 94.6 # 7
Question Answering SQuAD2.0 SpanBERT EM 85.7 # 66
F1 88.7 # 66
Question Answering SQuAD2.0 dev SpanBERT F1 86.8 # 6
Sentiment Analysis SST-2 Binary classification SpanBERT Accuracy 94.8 # 10
Semantic Textual Similarity STS Benchmark SpanBERT Pearson Correlation 0.899 # 8
Relation Extraction TACRED SpanBERT F1 70.8 # 2
Precision 70.8 # 1
Recall 70.9 # 1
Question Answering TriviaQA SpanBERT F1 83.6 # 1