Semantics-aware BERT for Language Understanding

5 Sep 2019  ·  Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, Xiang Zhou ·

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Natural Language Inference SNLI SemBERT % Test Accuracy 91.9 # 4
% Train Accuracy 94.4 # 16
Parameters 339m # 3
Question Answering SQuAD2.0 SemBERT(ensemble) EM 86.166 # 103
F1 88.886 # 108
Question Answering SQuAD2.0 SemBERT (single model) EM 84.800 # 130
F1 87.864 # 129
Question Answering SQuAD2.0 SemBERT (ensemble) EM 86.166 # 103
F1 88.886 # 108
Question Answering SQuAD2.0 dev SemBERT large F1 83.6 # 8
EM 80.9 # 7

Methods