Attention Modules

Edit

General • Attention • 42 methods

Attention Modules refer to modules that incorporate attention mechanisms. For example, multi-head attention is a module that incorporates multiple attention heads. Below you can find a continuously updating list of attention modules.

Methods

Add a Method

Method	Year	Papers
Multi-Head Attention Attention Is All You Need	2017	19930
Spatial Attention Module CBAM: Convolutional Block Attention Module	2018	188
SAGAN Self-Attention Module Self-Attention Generative Adversarial Networks	2018	136
Cross-Attention Module CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification	2021	100
Blender BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation	2020	80
Channel Attention Module CBAM: Convolutional Block Attention Module	2018	79
Graph Self-Attention BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	51
Slot Attention Object-Centric Learning with Slot Attention	2020	46
Deformable Attention Module Deformable DETR: Deformable Transformers for End-to-End Object Detection	2020	40
LAMA Low Rank Factorization for Compact Multi-Head Self-Attention	2019	37
SCA SCAM! Transferring humans between images with Semantic Cross Attention Modulation	2022	29
CBAM CBAM: Convolutional Block Attention Module	2018	27
Spatial-Reduction Attention Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions	2021	24
Multi-Head Linear Attention Linformer: Self-Attention with Linear Complexity	2020	18
Neighborhood Attention Neighborhood Attention Transformer	2022	15
Multi-DConv-Head Attention Primer: Searching for Efficient Transformers for Language Modeling	2021	12
Global Context Block GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond	2019	11
DV3 Attention Block Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning	2017	9
Bottleneck Transformer Block Bottleneck Transformers for Visual Recognition	2021	8
Triplet Attention Rotate to Attend: Convolutional Triplet Attention Module	2020	8
Spatial Attention-Guided Mask CenterMask : Real-Time Anchor-Free Instance Segmentation	2019	7
Single-Headed Attention Single Headed Attention RNN: Stop Thinking With Your Head	2019	5
Channel-wise Cross Attention UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer	2021	5
Re-Attention Module DeepViT: Towards Deeper Vision Transformer	2021	4
Mixed Attention Block ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	4
Attention-augmented Convolution Attention Augmented Convolutional Networks	2019	3
Point-wise Spatial Attention PSANet: Point-wise Spatial Attention Network for Scene Parsing	2018	3
Feedback Memory Addressing Some Limitations of Transformers with Feedback Memory	2020	3
Hopfield Layer Hopfield Networks is All You Need	2020	3
GPSA ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases	2021	3
Attention Free Transformer An Attention Free Transformer	2021	3
LeViT Attention Block LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference	2021	2
Spatially Separable Self-Attention Twins: Revisiting the Design of Spatial Attention in Vision Transformers	2021	2
Compact Global Descriptor Compact Global Descriptor for Neural Networks	2019	1
All-Attention Layer Augmenting Self-attention with Persistent Memory	2019	1
DeLighT Block DeLighT: Deep and Light-weight Transformer	2020	1
Peer-attention AssembleNet++: Assembling Modality Representations via Attention Connections	2020	1
Cross-Scale Non-Local Attention Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining	2020	1
Talking-Heads Attention Talking-Heads Attention	2020	1
SimAdapter Exploiting Adapters for Cross-lingual Low-resource Speech Recognition	2021	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1

Attention Modules Edit

Methods Add a Method

Attention Modules

Edit

Methods

Add a Method