Neural Semantic Encoders

EACL 2017  ·  Tsendsuren Munkhdalai, Hong Yu ·

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

PDF Abstract EACL 2017 PDF EACL 2017 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Natural Language Inference SNLI 300D MMA-NSE encoders with attention % Test Accuracy 85.4 # 72
% Train Accuracy 86.9 # 60
Parameters 3.2m # 4
Natural Language Inference SNLI 300D NSE encoders % Test Accuracy 84.6 # 77
% Train Accuracy 86.2 # 63
Parameters 3.0m # 4
Sentiment Analysis SST-2 Binary classification Neural Semantic Encoder Accuracy 89.7 # 62
Question Answering WikiQA MMA-NSE attention MAP 0.6811 # 18
MRR 0.6993 # 16
Machine Translation WMT2014 English-German NSE-NSE BLEU score 17.9 # 87
Hardware Burden None # 1
Operations per network pass None # 1

Methods


No methods listed for this paper. Add relevant methods here