Machine Comprehension Using Match-LSTM and Answer Pointer

29 Aug 2016  ·  Shuohang Wang, Jing Jiang ·

Machine comprehension of text is an important problem in natural language processing. A recently released dataset, the Stanford Question Answering Dataset (SQuAD), offers a large number of real questions and their answers created by humans through crowdsourcing. SQuAD provides a challenging testbed for evaluating machine comprehension algorithms, partly because compared with previous datasets, in SQuAD the answers do not come from a small set of candidate answers and they have variable lengths. We propose an end-to-end neural architecture for the task. The architecture is based on match-LSTM, a model we proposed previously for textual entailment, and Pointer Net, a sequence-to-sequence model proposed by Vinyals et al.(2015) to constrain the output tokens to be from the input sequences. We propose two ways of using Pointer Net for our task. Our experiments show that both of our two models substantially outperform the best results obtained by Rajpurkar et al.(2016) using logistic regression and manually crafted features.

PDF Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering SQuAD1.1 Match-LSTM with Ans-Ptr (Boundary) EM 60.474 # 186
F1 70.695 # 188
Hardware Burden None # 1
Operations per network pass None # 1
Question Answering SQuAD1.1 Match-LSTM with Ans-Ptr (Boundary) (ensemble) EM 67.901 # 166
F1 77.022 # 170
Question Answering SQuAD1.1 Match-LSTM with Ans-Ptr (Sentence) EM 54.505 # 190
F1 67.748 # 190
Hardware Burden None # 1
Operations per network pass None # 1
Question Answering SQuAD1.1 Match-LSTM with Bi-Ans-Ptr (Boundary) EM 64.744 # 176
F1 73.743 # 179
Hardware Burden None # 1
Operations per network pass None # 1
Question Answering SQuAD1.1 dev Match-LSTM with Bi-Ans-Ptr (Boundary+Search+b) EM 64.1 # 48
F1 64.7 # 53

Methods