Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

24 Feb 2020 • Alessandro Raganato • Yves Scherrer • Jörg Tiedemann

Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input... (read more)

PDF Abstract

Results from the Paper

Add Remove

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

METHOD	TYPE
Residual Connection	Skip Connections
BPE	Subword Segmentation
Dense Connections	Feedforward Networks
Label Smoothing	Regularization
ReLU	Activation Functions
Adam	Stochastic Optimization
Softmax	Output Functions
Dropout	Regularization
Multi-Head Attention	Attention Modules
Layer Normalization	Normalization
Scaled Dot-Product Attention	Attention Mechanisms
Transformer	Transformers

METHOD

TYPE

Residual Connection