Attention Modules

Gated Positional Self-Attention

Introduced by d'Ascoli et al. in ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

Gated Positional Self-Attention (GPSA) is a self-attention module for vision transformers, used in the ConViT architecture, that can be initialized as a convolutional layer -- helping a ViT learn inductive biases about locality.

Source: ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Image Classification 1 100.00%

Categories