Browse > Natural Language Processing > Question Answering > SQuAD1.1 dataset

Question Answering on SQuAD1.1
Add Remove

Rank Method EM F1 Paper title Year Paper Code
1
BERT (ensemble)
87.433 93.160
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2018
2
nlnet (ensemble)
85.954 91.677
3
nlnet (ensemble)
85.356 91.202
4
BERT (single model)
85.083 91.835
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2018
5
QANet (ensemble)
84.454 90.490
6
r-net (ensemble)
84.003 90.147
7
MARS (ensemble)
83.982 89.796
8
QANet (ensemble)
83.877 89.737
9
nlnet (single model)
83.468 90.133
10
MARS (single model)
83.185 89.547
11
MARS (single model)
83.122 89.224
12
Reinforced Mnemonic Reader + A2D (ensemble model)
82.849 88.764
13
QANet (ensemble)
82.744 89.045
14
r-net+ (ensemble)
82.650 88.493
15
MARS (single model)
82.587 88.880
16
Hybrid AoA Reader (ensemble)
82.482 89.281
17
QANet (single)
82.471 89.306
18
SLQA+ (ensemble)
82.440 88.607
19
Reinforced Mnemonic Reader (ensemble model)
82.283 88.533
Reinforced Mnemonic Reader for Machine Reading Comprehension
2017
20
QANet (single model)
82.209 88.608
21
r-net (ensemble)
82.136 88.126
22
AttentionReader+ (ensemble)
81.790 88.163
23
MMIPN
81.580 88.948
24
Reinforced Mnemonic Reader + A2D (single model)
81.538 88.130
25
KACTEIL-MRC(GF-Net+) (ensemble)
81.496 87.557
26
Reinforced Mnemonic Reader + A2D + DA (single model)
81.401 88.122
27
r-net (single model)
81.391 88.170
28
BiDAF + Self Attention + ELMo (ensemble)
81.003 87.432
Deep contextualized word representations
2018
29
QANet (single model)
80.929 87.773
30
Reinforced Mnemonic Reader + A2D (single model)
80.919 87.492
31
AVIQA+ (ensemble)
80.615 87.311
32
Reinforced Mnemonic Reader + A2D (single model)
80.489 87.454
33
SLQA+
80.436 87.021
34
{EAZI} (ensemble)
80.436 86.912
35
EAZI+ (ensemble)
80.426 86.912
36
DNET (ensemble)
80.164 86.721
37
Hybrid AoA Reader (single model)
80.027 87.288
38
BiDAF + Self Attention + ELMo + A2D (single model)
79.996 86.711
39
r-net+ (single model)
79.901 86.536
40
MAMCN+ (single model)
79.692 86.727
41
SAN (ensemble model)
79.608 86.496
Stochastic Answer Networks for Machine Reading Comprehension
2017
42
Reinforced Mnemonic Reader (single model)
79.545 86.654
Reinforced Mnemonic Reader for Machine Reading Comprehension
2017
43
SLQA+ (single model)
79.199 86.590
44
Interactive AoA Reader+ (ensemble)
79.083 86.450
45
MIR-MRC(F-Net) (single model)
79.083 86.288
46
MDReader
79.031 86.006
47
FusionNet (ensemble)
78.978 86.016
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
2017
48
DCN+ (ensemble)
78.852 85.996
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
2017
49
KACTEIL-MRC(GF-Net+) (single model)
78.664 85.780
50
BiDAF + Self Attention + ELMo (single model)
78.580 85.833
Deep contextualized word representations
2018
51
aviqa (ensemble)
78.496 85.469
52
Conductor-net (ensemble)
78.433 85.517
Phase Conductor on Multi-layered Attentions for Machine Comprehension
2017
53
KakaoNet (single model)
78.401 85.724
54
SLQA(ensemble)
78.328 85.682
55
MEMEN (single model)
78.234 85.344
MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
2017
56
BiDAF++ with pair2vec (single model)
78.223 85.535
57
MDReader0
78.171 85.543
58
test
78.087 85.348
59
Interactive AoA Reader (ensemble)
77.845 85.297
60
DNET (single model)
77.646 84.905
61
RaSoR + TR + LM (single model)
77.583 84.163
Contextualized Word Representations for Reading Comprehension
2017
62
BiDAF++ (single model)
77.573 84.858
63
AttentionReader+ (single)
77.342 84.925
64
Jenga (ensemble)
77.237 84.466
65
{gqa} (single model)
77.090 83.931
66
Conductor-net (ensemble)
76.996 84.630
Phase Conductor on Multi-layered Attentions for Machine Comprehension
2017
67
MARS (single model)
76.859 84.739
68
SAN (single model)
76.828 84.396
Stochastic Answer Networks for Machine Reading Comprehension
2017
69
VS^3-NET (single model)
76.775 84.491
70
r-net (single model)
76.461 84.265
Gated Self-Matching Networks for Reading Comprehension and Question Answering
71
FRC (single model)
76.240 84.599
72
QANet + data augmentation ×3
76.2 84.6
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
2018
73
Conductor-net (ensemble)
76.146 83.991
74
KAR (single model)
76.125 83.538
Exploring Machine Reading Comprehension with Explicit Knowledge
2018
75
smarnet (ensemble)
75.989 83.475
76
FusionNet (single model)
75.968 83.900
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
2017
77
AVIQA-v2 (single model)
75.926 83.305
78
Interactive AoA Reader+ (single model)
75.821 83.843
79
RaSoR + TR (single model)
75.789 83.261
Contextualized Word Representations for Reading Comprehension
2017
80
MEMEN (ensemble)
75.370 82.658
MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
2017
81
Mixed model (ensemble)
75.265 82.769
82
two-attention-self-attention (ensemble)
75.223 82.716
83
DCN+ (single model)
75.087 83.081
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
2017
84
ReasoNet (ensemble)
75.034 82.552
ReasoNet: Learning to Stop Reading in Machine Comprehension
2016
85
DCN+ (single model)
74.866 82.806
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
2017
86
eeAttNet (single model)
74.604 82.501
87
SSR-BiDAF
74.541 82.477
88
SLQA (single model)
74.489 82.815
89
Conductor-net (single model)
74.405 82.742
Phase Conductor on Multi-layered Attentions for Machine Comprehension
2017
90
Jenga (single model)
74.373 82.845
91
Mnemonic Reader (ensemble)
74.268 82.371
Reinforced Mnemonic Reader for Machine Reading Comprehension
2017
92
S^3-Net (ensemble)
74.121 82.342
93
SEDT (ensemble model)
74.090 81.761
Structural Embedding of Syntactic Trees for Machine Comprehension
2017
94
SSAE (ensemble)
74.080 81.665
95
Multi-Perspective Matching (ensemble)
73.765 81.257
Multi-Perspective Context Matching for Machine Comprehension
2016
96
BiDAF (ensemble)
73.744 81.525
Bidirectional Attention Flow for Machine Comprehension
2016
97
SEDT+BiDAF (ensemble)
73.723 81.530
Structural Embedding of Syntactic Trees for Machine Comprehension
2017
98
Interactive AoA Reader (single model)
73.639 81.931
99
Jenga (single model)
73.303 81.754
100
Conductor-net (single)
73.240 81.933
Phase Conductor on Multi-layered Attentions for Machine Comprehension
2017
101
jNet (ensemble)
73.010 81.517
Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
2017
102
T-gating (ensemble)
72.758 81.001
103
two-attention-self-attention (single model)
72.600 81.011
104
Conductor-net (single)
72.590 81.415
105
AVIQA (single model)
72.485 80.550
106
BiDAF + Self Attention (single model)
72.139 81.048
Simple and Effective Multi-Paragraph Reading Comprehension
2017
107
S^3-Net (single model)
71.908 81.023
108
QFASE
71.898 79.989
109
attention+self-attention (single model)
71.698 80.462
110
Dynamic Coattention Networks (ensemble)
71.625 80.383
Dynamic Coattention Networks For Question Answering
2016
111
smarnet (single model)
71.415 80.160
Smarnet: Teaching Machines to Read and Comprehend Like Human
2017
112
SRU
71.4 80.2
Simple Recurrent Units for Highly Parallelizable Recurrence
2017
113
AttReader (single)
71.373 79.725
114
DCN + Char + CoVe
71.3 79.9
Learned in Translation: Contextualized Word Vectors
2017
115
M-NET (single)
71.016 79.835
116
Mnemonic Reader (single model)
70.995 80.146
Reinforced Mnemonic Reader for Machine Reading Comprehension
2017
117
MAMCN (single model)
70.985 79.939
118
RaSoR (single model)
70.849 78.741
Learning Recurrent Span Representations for Extractive Question Answering
2016
119
FastQAExt
70.849 78.857
Making Neural QA as Simple as Possible but not Simpler
2017
120
Document Reader (single model)
70.733 79.353
Reading Wikipedia to Answer Open-Domain Questions
2017
121
Ruminating Reader (single model)
70.639 79.456
Ruminating Reader: Reasoning with Gated Multi-Hop Attention
2017
122
jNet (single model)
70.607 79.821
Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
2017
123
ReasoNet (single model)
70.555 79.364
ReasoNet: Learning to Stop Reading in Machine Comprehension
2016
124
Multi-Perspective Matching (single model)
70.387 78.784
Multi-Perspective Context Matching for Machine Comprehension
2016
125
SimpleBaseline (single model)
69.600 78.236
126
SSR-BiDAF
69.443 78.358
127
SEDT+BiDAF (single model)
68.478 77.971
Structural Embedding of Syntactic Trees for Machine Comprehension
2017
128
FastQA
68.436 77.070
Making Neural QA as Simple as Possible but not Simpler
2017
129
PQMN (single model)
68.331 77.783
130
SEDT (single model)
68.163 77.527
Structural Embedding of Syntactic Trees for Machine Comprehension
2017
131
T-gating (single model)
68.132 77.569
132
BiDAF (single model)
67.974 77.323
Bidirectional Attention Flow for Machine Comprehension
2016
133
Match-LSTM with Ans-Ptr (Boundary) (ensemble)
67.901 77.022
Machine Comprehension Using Match-LSTM and Answer Pointer
2016
134
FABIR
67.744 77.605
A Fully Attention-Based Information Retriever
2018
135
AllenNLP BiDAF (single model)
67.618 77.151
136
Iterative Co-attention Network
67.502 76.786
137
newtest
66.527 75.787
138
Dynamic Coattention Networks (single model)
66.233 75.896
Dynamic Coattention Networks For Question Answering
2016
139
Match-LSTM with Bi-Ans-Ptr (Boundary)
64.744 73.743
Machine Comprehension Using Match-LSTM and Answer Pointer
2016
140
Unnamed submission by ravioncodalab
64.439 73.921
141
OTF dict+spelling (single)
64.083 73.056
Learning to Compute Word Embeddings On the Fly
2017
142
Attentive CNN context with LSTM
63.306 73.463
143
OTF spelling (single)
62.897 72.016
Learning to Compute Word Embeddings On the Fly
2017
144
OTF spelling+lemma (single)
62.604 71.968
Learning to Compute Word Embeddings On the Fly
2017
145
Dynamic Chunk Reader
62.499 70.956
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
2016
146
Fine-Grained Gating
62.446 73.327
Words or Characters? Fine-grained Gating for Reading Comprehension
2016
147
Match-LSTM with Ans-Ptr (Boundary)
60.474 70.695
Machine Comprehension Using Match-LSTM and Answer Pointer
2016
148
Unnamed submission by Will_Wu
59.058 69.436
149
Match-LSTM with Ans-Ptr (Sentence)
54.505 67.748
Machine Comprehension Using Match-LSTM and Answer Pointer
2016
150
Unnamed submission by jinhyuklee
52.544 62.780
151
Unnamed submission by minjoon
52.533 62.757
Contact us on: [email protected]. Papers With Code is a free resource supported by Atlas ML.