On the Relationship between Self-Attention and Convolutional Layers

8 Nov 2019Jean-Baptiste CordonnierAndreas LoukasMartin Jaggi

Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks... (read more)

PDF Abstract

Evaluation Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK COMPARE
Image Classification CIFAR-10 SA quadratic emb. Percentage correct 93.8 # 37