Attention Mechanisms

Edit

General • Attention • 82 methods

Attention Mechanisms are a component used in neural networks to model long-range interaction, for example across a text in NLP. The key idea is to build shortcuts between a context vector and the input, to allow a model to attend to different parts. Below you can find a continuously updating list of attention mechanisms.

Subcategories

1 Attention Patterns

2 Synthesized Attention Mechanisms

Methods

Add a Method

Method	Year	Papers
Scaled Dot-Product Attention Attention Is All You Need	2017	17786
Strided Attention Generating Long Sequences with Sparse Transformers	2019	1390
Fixed Factorized Attention Generating Long Sequences with Sparse Transformers	2019	1389
RAN Residual Attention Network for Image Classification	2017	264
Temporal attention Jointly Attentive Spatial-Temporal Pooling Networks for Video-based Person Re-Identification	2017	221
Dot-Product Attention Effective Approaches to Attention-based Neural Machine Translation	2015	219
Additive Attention Neural Machine Translation by Jointly Learning to Align and Translate	2014	196
Channel attention Squeeze-and-Excitation Networks	2017	172
SPIN Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations	2022	117
RAM Recurrent Models of Visual Attention	2014	106
Sliding Window Attention Longformer: The Long-Document Transformer	2020	85
Global and Sliding Window Attention Longformer: The Long-Document Transformer	2020	79
Dilated Sliding Window Attention Longformer: The Long-Document Transformer	2020	78
FAVOR+ Rethinking Attention with Performers	2020	73
Dynamic Convolution Dynamic Convolution: Attention over Convolution Kernels	2019	67
Disentangled Attention Mechanism DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	66
Axial Attention Axial Attention in Multidimensional Transformers	2019	52
Rendezvous Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	46
Global-Local Attention ETC: Encoding Long and Structured Inputs in Transformers	2020	43
Attention Gate Attention U-Net: Learning Where to Look for the Pancreas	2018	39
Channel-wise Soft Attention	2017	38
STN Spatial Transformer Networks	2015	33
Content-based Attention Neural Turing Machines	2014	32
Class Attention Going deeper with Image Transformers	2021	32
Location-based Attention Effective Approaches to Attention-based Neural Machine Translation	2015	31
Location Sensitive Attention Attention-Based Models for Speech Recognition	2015	24
TAM TAM: Temporal Adaptive Module for Video Recognition	2020	24
BAM BAM: Bottleneck Attention Module	2018	24
SRM SRM: A Style-Based Recalibration Module for Convolutional Neural Networks	2019	23
Coordinate attention Coordinate Attention for Efficient Mobile Network Design	2021	21
Adaptive Masking Adaptive Attention Span in Transformers	2019	20
Highway networks Highway Networks	2015	19
DMA Text-Guided Neural Image Inpainting	2020	19
LSH Attention Reformer: The Efficient Transformer	2020	17
Set Transformer Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks	2018	16
BigBird Big Bird: Transformers for Longer Sequences	2020	15
Neighborhood Attention Neighborhood Attention Transformer	2022	14
SEAM Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation	2020	11
Bi-attention Bilinear Attention Networks for Person Retrieval	2019	9
DANet Dual Attention Network for Scene Segmentation	2018	9
RGA Relation-Aware Global Attention for Person Re-identification	2019	7
Routing Attention Efficient Content-Based Sparse Attention with Routing Transformers	2020	7
GCT Gated Channel Transformation for Visual Recognition	2019	6
GALA Learning what and where to attend	2018	6
Multiplicative Attention Effective Approaches to Attention-based Neural Machine Translation	2015	6
Cross-Covariance Attention XCiT: Cross-Covariance Image Transformers	2021	5
scSE Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks	2018	4
SPNet Strip Pooling: Rethinking Spatial Pooling for Scene Parsing	2020	4
Channel Squeeze and Spatial Excitation Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks	2018	3
Spatial & Temporal Attention An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data	2016	3
Branch attention Training Very Deep Networks	2015	3
3D SA Attention based Writer Independent Handwriting Verification	2020	3
Deformable ConvNets Deformable Convolutional Networks	2017	2
Sparse Sinkhorn Attention Sparse Sinkhorn Attention	2020	2
SCA-CNN SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning	2016	2
Concurrent Spatial and Channel Squeeze & Excitation Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks	2018	2
Global Sub-Sampled Attention Twins: Revisiting the Design of Spatial Attention in Vision Transformers	2021	2
FcaNet FcaNet: Frequency Channel Attention Networks	2020	2
Channel & Spatial attention Residual Attention Network for Image Classification	2017	2
Locally-Grouped Self-Attention Twins: Revisiting the Design of Spatial Attention in Vision Transformers	2021	2
ECANet ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks	2019	2
Self-Calibrated Convolutions Improving Convolutional Networks With Self-Calibrated Convolutions	2020	2
Class Activation Guided Attention Mechanism Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	2
Factorized Dense Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
HyperHyperNetwork HyperHyperNetworks for the Design of Antenna Arrays	2021	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
Random Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
Gather-Excite Networks Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks	2018	1
GSoP-Net Global Second-order Pooling Convolutional Networks	2018	1
Dense Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
Attention Feature Filters NeuriCam: Key-Frame Video Super-Resolution and Colorization for IoT Cameras	2022	1
STA-LSTM An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data	2016	1
HyperSA Hypergraph Transformer for Skeleton-based Action Recognition	2022	1
SortCut Sinkhorn Attention Sparse Sinkhorn Attention	2020	1
Segregated Attention Network InferNER: an attentive model leveraging the sentence-level information for Named Entity Recognition in Microblogs	2021	1
Fast Voxel Query Voxel Transformer for 3D Object Detection	2021	1
ProCAN ProCAN: Progressive Growing Channel Attentive Non-Local Network for Lung Nodule Classification	2020	1
Factorized Random Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
Differential attention for visual question answering	2000	0

Attention Mechanisms Edit

Methods Add a Method

Attention Mechanisms

Edit

Methods

Add a Method