Autoregressive Transformers


Introduced by Vaswani et al. in Attention Is All You Need

A Transformer is a model architecture that eschews recurrence and instead relies entirely on an attention mechanism to draw global dependencies between input and output. Before Transformers, the dominant sequence transduction models were based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The Transformer also employs an encoder and decoder, but removing recurrence in favor of attention mechanisms allows for significantly more parallelization than methods like RNNs and CNNs.

Source: Attention Is All You Need


Paper Code Results Date Stars


Task Papers Share
Language Modelling 63 8.73%
Semantic Segmentation 29 4.02%
Retrieval 25 3.46%
Question Answering 24 3.32%
Text Generation 24 3.32%
Speech Recognition 18 2.49%
Image Segmentation 15 2.08%
Machine Translation 13 1.80%
Decision Making 12 1.66%