Methods > General > Attention

Attention Mechanisms

Attention Mechanisms are a component used in neural networks to model long-range interaction, for example across a text in NLP. The key idea is to build shortcuts between a context vector and the input, to allow a model to attend to different parts. Below you can find a continuously updating list of attention mechanisms.

METHOD YEAR PAPERS
Scaled Dot-Product Attention
2017 3851
Additive Attention
2014 110
Dot-Product Attention
2015 75
Strided Attention
2019 41
Fixed Factorized Attention
2019 41
Content-based Attention
2014 25
Location-based Attention
2015 24
Global and Sliding Window Attention
2020 14
Dilated Sliding Window Attention
2020 14
Sliding Window Attention
2020 14
Channel-wise Soft Attention
2017 14
Location Sensitive Attention
2015 14
LSH Attention
2020 8
Set Transformer
2018 7
Adaptive Masking
2019 4
Multiplicative Attention
2015 2
Routing Attention
2020 2
Factorized Dense Synthesized Attention
2020 1
Random Synthesized Attention
2020 1
Dense Synthesized Attention
2020 1
Factorized Random Synthesized Attention
2020 1
Sparse Sinkhorn Attention
2020 1
SortCut Sinkhorn Attention
2020 1
3D SA
2020 1
DMA
2020 1
ProCAN
2020 1
Differential attention for visual question answering
2000 0