R-Drop: Regularized Dropout for Neural Networks

Dropout is a powerful and widely used technique to regularize the training of deep neural networks. In this paper, we introduce a simple regularization strategy upon dropout in model training, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the freedom of the model parameters and complements dropout. Experiments on $\bf{5}$ widely used deep learning tasks ($\bf{18}$ datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English$\to$German translation ($\bf{30.91}$ BLEU) and WMT14 English$\to$French translation ($\bf{43.95}$ BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub{\url{https://github.com/dropreg/R-Drop}}.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Abstractive Text Summarization CNN / Daily Mail BART + R-Drop ROUGE-1 44.51 # 10
ROUGE-2 21.58 # 6
ROUGE-L 41.24 # 16
Machine Translation IWSLT2014 German-English Transformer + R-Drop + Cutoff BLEU score 37.90 # 5
Machine Translation IWSLT2014 German-English Transformer + R-Drop BLEU score 37.25 # 10
Machine Translation WMT2014 English-French Transformer + R-Drop BLEU score 43.95 # 4
Hardware Burden None # 1
Operations per network pass None # 1
Machine Translation WMT2014 English-German Transformer + R-Drop BLEU score 30.91 # 6
Hardware Burden 49G # 1
Operations per network pass None # 1