Introduced by Vaswani et al. in Attention Is All You Need

A Transformer is a model architecture that eschews recurrence and instead relies entirely on an attention mechanism to draw global dependencies between input and output. Before Transformers, the dominant sequence transduction models were based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The Transformer also employs an encoder and decoder, but removing recurrence in favor of attention mechanisms allows for significantly more parallelization than methods like RNNs and CNNs.

Source: Attention Is All You Need


Paper Code Results Date Stars


Task Papers Share
Language Modelling 44 6.18%
Decoder 24 3.37%
Large Language Model 22 3.09%
Question Answering 20 2.81%
In-Context Learning 19 2.67%
Retrieval 16 2.25%
Benchmarking 14 1.97%
Decision Making 14 1.97%
Time Series Forecasting 12 1.69%