Attention Mechanisms

Edit

General • Attention • 86 methods

Attention Mechanisms are a component used in neural networks to model long-range interaction, for example across a text in NLP. The key idea is to build shortcuts between a context vector and the input, to allow a model to attend to different parts. Below you can find a continuously updating list of attention mechanisms.

Subcategories

1 Attention Patterns

2 Synthesized Attention Mechanisms

Methods

Add a Method

Method	Year	Papers
Scaled Dot-Product Attention Attention Is All You Need	2017	22497
Fixed Factorized Attention Generating Long Sequences with Sparse Transformers	2019	1793
Strided Attention Generating Long Sequences with Sparse Transformers	2019	1792
VAE Auto-Encoding Variational Bayes	2013	1200
RAN Residual Attention Network for Image Classification	2017	314
Temporal attention Jointly Attentive Spatial-Temporal Pooling Networks for Video-based Person Re-Identification	2017	285
Channel attention Squeeze-and-Excitation Networks	2017	239
Dot-Product Attention Effective Approaches to Attention-based Neural Machine Translation	2015	237
Additive Attention Neural Machine Translation by Jointly Learning to Align and Translate	2014	203
SPIN Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations	2022	157
RAM Recurrent Models of Visual Attention	2014	138
Sliding Window Attention Longformer: The Long-Document Transformer	2020	95
FAVOR+ Rethinking Attention with Performers	2020	93
Global and Sliding Window Attention Longformer: The Long-Document Transformer	2020	83
Disentangled Attention Mechanism DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	82
Dilated Sliding Window Attention Longformer: The Long-Document Transformer	2020	82
Dynamic Convolution Dynamic Convolution: Attention over Convolution Kernels	2019	81
Rendezvous Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	67
Axial Attention Axial Attention in Multidimensional Transformers	2019	57
Global-Local Attention ETC: Encoding Long and Structured Inputs in Transformers	2020	52
Channel-wise Soft Attention	2017	48
Attention Gate Attention U-Net: Learning Where to Look for the Pancreas	2018	43
STN Spatial Transformer Networks	2015	39
Class Attention Going deeper with Image Transformers	2021	35
Location-based Attention Effective Approaches to Attention-based Neural Machine Translation	2015	35
Content-based Attention Neural Turing Machines	2014	32
DMA Text-Guided Neural Image Inpainting	2020	32
BAM BAM: Bottleneck Attention Module	2018	31
SRM SRM: A Style-Based Recalibration Module for Convolutional Neural Networks	2019	30
TAM TAM: Temporal Adaptive Module for Video Recognition	2020	30
Coordinate attention Coordinate Attention for Efficient Mobile Network Design	2021	28
Location Sensitive Attention Attention-Based Models for Speech Recognition	2015	25
Adaptive Masking Adaptive Attention Span in Transformers	2019	24
Highway networks Highway Networks	2015	22
LSH Attention Reformer: The Efficient Transformer	2020	20
Neighborhood Attention Neighborhood Attention Transformer	2022	18
Set Transformer Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks	2018	17
BigBird Big Bird: Transformers for Longer Sequences	2020	16
SEAM Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation	2020	13
Bi-attention Bilinear Attention Networks for Person Retrieval	2019	11
Routing Attention Efficient Content-Based Sparse Attention with Routing Transformers	2020	11
DANet Dual Attention Network for Scene Segmentation	2018	10
RGA Relation-Aware Global Attention for Person Re-identification	2019	8
GALA Learning what and where to attend	2018	8
Cross-Covariance Attention XCiT: Cross-Covariance Image Transformers	2021	7
GCT Gated Channel Transformation for Visual Recognition	2019	7
Multiplicative Attention Effective Approaches to Attention-based Neural Machine Translation	2015	6
SPNet Strip Pooling: Rethinking Spatial Pooling for Scene Parsing	2020	5
scSE Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks	2018	5
ECANet ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks	2019	5
Channel Squeeze and Spatial Excitation Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks	2018	3
Spatial & Temporal Attention An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data	2016	3
Branch attention Training Very Deep Networks	2015	3
3D SA Attention based Writer Independent Handwriting Verification	2020	3
Deformable ConvNets Deformable Convolutional Networks	2017	2
Sparse Sinkhorn Attention Sparse Sinkhorn Attention	2020	2
SCA-CNN SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning	2016	2
Concurrent Spatial and Channel Squeeze & Excitation Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks	2018	2
Global Sub-Sampled Attention Twins: Revisiting the Design of Spatial Attention in Vision Transformers	2021	2
FcaNet FcaNet: Frequency Channel Attention Networks	2020	2
Channel & Spatial attention Residual Attention Network for Image Classification	2017	2
Locally-Grouped Self-Attention Twins: Revisiting the Design of Spatial Attention in Vision Transformers	2021	2
Self-Calibrated Convolutions Improving Convolutional Networks With Self-Calibrated Convolutions	2020	2
Class Activation Guided Attention Mechanism Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	2
STA-LSTM An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data	2016	2
Factorized Dense Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
HyperHyperNetwork HyperHyperNetworks for the Design of Antenna Arrays	2021	1
Multidimensional Convolutional Frequency Attention	2000	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
Random Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
Gather-Excite Networks Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks	2018	1
GSoP-Net Global Second-order Pooling Convolutional Networks	2018	1
Dense Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
Vision Eagle Attention Vision Eagle Attention: a new lens for advancing image classification	2024	1
Attention Feature Filters NeuriCam: Key-Frame Video Super-Resolution and Colorization for IoT Cameras	2022	1
HyperSA Hypergraph Transformer for Skeleton-based Action Recognition	2022	1
SortCut Sinkhorn Attention Sparse Sinkhorn Attention	2020	1
Segregated Attention Network InferNER: an attentive model leveraging the sentence-level information for Named Entity Recognition in Microblogs	2021	1
Fast Voxel Query Voxel Transformer for 3D Object Detection	2021	1
ProCAN ProCAN: Progressive Growing Channel Attentive Non-Local Network for Lung Nodule Classification	2020	1
Factorized Random Synthesized Attention Synthesizer: Rethinking Self-Attention in Transformer Models	2020	1
Differential attention for visual question answering	2000	0
Spatial Attention ConvMixer	2000	0

Attention Mechanisms Edit

Methods Add a Method

Attention Mechanisms

Edit

Methods

Add a Method