Depth Growing for Neural Machine Translation

While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even reduces performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT$14$ English$\to$German and English$\to$French translation tasks\footnote{Our code is available at \url{https://github.com/apeterswu/Depth_Growing_NMT}}.

PDF Abstract ACL 2019 PDF ACL 2019 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Machine Translation WMT2014 English-French Depth Growing BLEU score 43.27 # 11
Hardware Burden 24G # 1
Operations per network pass None # 1
Machine Translation WMT2014 English-German Depth Growing BLEU score 30.07 # 11
Hardware Burden 24G # 1
Operations per network pass None # 1

Methods