Autoregressive Transformers


Introduced by Wang et al. in Linformer: Self-Attention with Linear Complexity

Linformer is a linear Transformer that utilises a linear self-attention mechanism to tackle the self-attention bottleneck with Transformer models. The original scaled dot-product attention is decomposed into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention.

Source: Linformer: Self-Attention with Linear Complexity


Paper Code Results Date Stars


Task Papers Share
Image Classification 1 16.67%
Image Generation 1 16.67%
Autonomous Driving 1 16.67%
Depth Estimation 1 16.67%
Knowledge Distillation 1 16.67%
Language Modelling 1 16.67%