Transformers

Transformer

Introduced by Vaswani et al. in Attention Is All You Need

A Transformer is a model architecture that eschews recurrence and instead relies entirely on an attention mechanism to draw global dependencies between input and output. Before Transformers, the dominant sequence transduction models were based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The Transformer also employs an encoder and decoder, but removing recurrence in favor of attention mechanisms allows for significantly more parallelization than methods like RNNs and CNNs.

Source: Attention Is All You Need

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Language Modelling 64 8.25%
Large Language Model 32 4.12%
In-Context Learning 19 2.45%
Retrieval 19 2.45%
Machine Translation 19 2.45%
Question Answering 18 2.32%
Semantic Segmentation 16 2.06%
Decision Making 16 2.06%
Object Detection 15 1.93%

Categories