Mask Attention Networks: Rethinking and Strengthen Transformer

Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.

PDF Abstract NAACL 2021 PDF NAACL 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Abstractive Text Summarization CNN / Daily Mail Mask Attention Network ROUGE-1 40.98 # 29
ROUGE-2 18.29 # 29
ROUGE-L 37.88 # 28
Text Summarization GigaWord Mask Attention Network ROUGE-1 38.28 # 16
ROUGE-2 19.46 # 16
ROUGE-L 35.46 # 16
Machine Translation IWSLT2014 German-English Mask Attention Network (small) BLEU score 36.3 # 11
Number of Params 37M # 1
Machine Translation WMT2014 English-German Mask Attention Network (big) BLEU score 30.4 # 11
Number of Params 215M # 1
Hardware Burden None # 1
Operations per network pass None # 1
Machine Translation WMT2014 English-German Mask Attention Network (base) BLEU score 29.1 # 30
Number of Params 63M # 2
Hardware Burden None # 1
Operations per network pass None # 1

Methods