Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling

13 Nov 2020  ·  Shruti Bhosale, Kyra Yee, Sergey Edunov, Michael Auli ·

Pre-training models on vast quantities of unlabeled data has emerged as an effective approach to improving accuracy on many NLP tasks. On the other hand, traditional machine translation has a long history of leveraging unlabeled data through noisy channel modeling... The same idea has recently been shown to achieve strong improvements for neural machine translation. Unfortunately, na\"{i}ve noisy channel modeling with modern sequence to sequence models is up to an order of magnitude slower than alternatives. We address this issue by introducing efficient approximations to make inference with the noisy channel approach as fast as strong ensembles while increasing accuracy. We also show that the noisy channel approach can outperform strong pre-training results by achieving a new state of the art on WMT Romanian-English translation. read more

PDF Abstract

Datasets


Results from the Paper


 Ranked #1 on Machine Translation on WMT2016 Romanian-English (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Machine Translation WMT2016 Romanian-English fast-noisy-channel-modeling BLEU score 40.3 # 1

Methods


No methods listed for this paper. Add relevant methods here