Improving Neural Language Modeling via Adversarial Training

10 Jun 2019Dilin WangChengyue GongQiang Liu

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting... (read more)

PDF Abstract

Code


No code implementations yet. Submit your code now

Evaluation Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK COMPARE
Machine Translation IWSLT2015 German-English Transformer Base + adversarial MLE BLEU score 35.18 # 1
Language Modelling Penn Treebank (Word Level) adversarial + AWD-LSTM-MoS + dynamic eval Validation perplexity 46.63 # 1
Language Modelling Penn Treebank (Word Level) adversarial + AWD-LSTM-MoS + dynamic eval Test perplexity 46.01 # 2
Language Modelling Penn Treebank (Word Level) adversarial + AWD-LSTM-MoS + dynamic eval Params 22M # 1
Language Modelling WikiText-2 adversarial + AWD-LSTM-MoS + dynamic eval Validation perplexity 40.27 # 1
Language Modelling WikiText-2 adversarial + AWD-LSTM-MoS + dynamic eval Test perplexity 38.65 # 2
Language Modelling WikiText-2 adversarial + AWD-LSTM-MoS + dynamic eval Number of params 35M # 1
Machine Translation WMT2014 English-German Transformer Big + adversarial MLE BLEU score 29.52 # 8