Learned in Translation: Contextualized Word Vectors

NeurIPS 2017 Bryan McCannJames BradburyCaiming XiongRichard Socher

Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors... (read more)

PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT LEADERBOARD
Sentiment Analysis IMDb BCN+Char+CoVe Accuracy 91.8 # 12
Natural Language Inference SNLI Biattentive Classification Network + CoVe + Char % Test Accuracy 88.1 # 17
% Train Accuracy 88.5 # 42
Parameters 22m # 2
Question Answering SQuAD1.1 DCN + Char + CoVe EM 71.3 # 128
F1 79.9 # 131
Question Answering SQuAD1.1 dev DCN (Char + CoVe) EM 71.3 # 23
F1 79.9 # 27
Sentiment Analysis SST-2 Binary classification BCN+Char+CoVe Accuracy 90.3 # 22
Sentiment Analysis SST-5 Fine-grained classification BCN+Char+CoVe Accuracy 53.7 # 4
Text Classification TREC-6 CoVe Error 4.2 # 6

Methods used in the Paper