Stochastic Answer Networks for Machine Reading Comprehension

ACL 2018  ·  Xiaodong Liu, Yelong Shen, Kevin Duh, Jianfeng Gao ·

We propose a simple yet robust stochastic answer network (SAN) that simulates multi-step reasoning in machine reading comprehension. Compared to previous work such as ReasoNet which used reinforcement learning to determine the number of steps, the unique feature is the use of a kind of stochastic prediction dropout on the answer module (final layer) of the neural network during the training. We show that this simple trick improves robustness and achieves results competitive to the state-of-the-art on the Stanford Question Answering Dataset (SQuAD), the Adversarial SQuAD, and the Microsoft MAchine Reading COmprehension Dataset (MS MARCO).

PDF Abstract ACL 2018 PDF ACL 2018 Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering SQuAD1.1 SAN (single model) EM 76.828 # 102
F1 84.396 # 105
Hardware Burden None # 1
Operations per network pass None # 1
Question Answering SQuAD1.1 SAN (ensemble model) EM 79.608 # 68
F1 86.496 # 73
Hardware Burden None # 1
Operations per network pass None # 1
Question Answering SQuAD1.1 dev SAN (single) EM 76.235 # 25
F1 84.056 # 28
Question Answering SQuAD2.0 SAN (ensemble model) EM 71.316 # 238
F1 73.704 # 242
Question Answering SQuAD2.0 SAN (single model) EM 68.653 # 246
F1 71.439 # 249

Methods


No methods listed for this paper. Add relevant methods here