SG-Net: Syntax-Guided Machine Reading Comprehension

14 Aug 2019  ·  Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, Rui Wang ·

For machine reading comprehension, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy passages and getting ride of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. To verify its effectiveness, the proposed SG-Net is applied to typical pre-trained language model BERT which is right based on a Transformer encoder. Extensive experiments on popular benchmarks including SQuAD 2.0 and RACE show that the proposed SG-Net design helps achieve substantial performance improvement over strong baselines.

PDF Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering SQuAD2.0 XLNet + SG-Net Verifier (ensemble) EM 88.174 # 63
F1 90.702 # 77
Question Answering SQuAD2.0 SG-Net (single model) EM 85.229 # 122
F1 87.926 # 128
Question Answering SQuAD2.0 XLNet + SG-Net Verifier++ (single model) EM 87.238 # 85
F1 90.071 # 87
Question Answering SQuAD2.0 SG-Net (ensemble) EM 86.211 # 102
F1 88.848 # 114
Question Answering SQuAD2.0 dev SG-Net F1 87.9 # 5
EM 85.1 # 4