Rethinking Perturbations in Encoder-Decoders for Fast Training

NAACL 2021  ·  Sho Takase, Shun Kiyono ·

We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations.

PDF Abstract NAACL 2021 PDF NAACL 2021 Abstract

Results from the Paper


 Ranked #1 on Text Summarization on DUC 2004 Task 1 (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Text Summarization DUC 2004 Task 1 Transformer+WDrop ROUGE-1 33.06 # 1
ROUGE-2 11.45 # 4
ROUGE-L 28.51 # 2
Text Summarization GigaWord Transformer+Rep(Uni) ROUGE-1 39.81 # 3
ROUGE-2 20.40 # 7
ROUGE-L 36.93 # 2
Text Summarization GigaWord Transformer+Wdrop ROUGE-1 39.66 # 5
ROUGE-2 20.45 # 5
ROUGE-L 36.59 # 8
Machine Translation IWSLT2014 German-English Transformer+Rep(Sim)+WDrop BLEU score 36.22 # 15
Number of Params 37M # 1
Machine Translation WMT2014 English-German Transformer+Rep(Uni) BLEU score 33.89 # 2
SacreBLEU 32.35 # 2
Hardware Burden None # 1
Operations per network pass None # 1

Methods