SpanBERT: Improving Pre-training by Representing and Predicting Spans

24 Jul 2019Mandar JoshiDanqi ChenYinhan LiuDaniel S. WeldLuke ZettlemoyerOmer Levy

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it... (read more)

PDF Abstract

Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Question Answering HotpotQA SpanBERT Joint F1 83.0 # 1
Question Answering NaturalQA SpanBERT F1 82.5 # 1
Question Answering NewsQA SpanBERT F1 73.6 # 1
Open-Domain Question Answering SearchQA SpanBERT F1 84.8 # 1
Question Answering SQuAD1.1 SpanBERT EM 88.8 # 2
Question Answering SQuAD1.1 SpanBERT F1 94.6 # 2
Question Answering SQuAD2.0 SpanBERT EM 85.7 # 20
Question Answering SQuAD2.0 SpanBERT F1 88.7 # 17
Relation Extraction TACRED SpanBERT F1 70.8 # 2