ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

19 Mar 2021  ·  Stéphane d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, Levent Sagun ·

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ``soft" convolutional inductive bias. We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analysing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at

PDF Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Classification ImageNet ConViT-B+ Top 1 Accuracy 82.5% # 314
Number of params 152M # 636
Hardware Burden None # 1
Operations per network pass None # 1
GFLOPs 30 # 308
Image Classification ImageNet ConViT-B Top 1 Accuracy 82.4% # 321
Number of params 86M # 586
GFLOPs 17 # 283
Image Classification ImageNet ConViT-S+ Top 1 Accuracy 82.2% # 334
Number of params 48M # 503
GFLOPs 10 # 248
Image Classification ImageNet ConViT-S Top 1 Accuracy 81.3% # 398
Number of params 27M # 429
GFLOPs 5.4 # 197
Image Classification ImageNet ConViT-Ti+ Top 1 Accuracy 76.7% # 586
Number of params 10M # 319
GFLOPs 2 # 125
Image Classification ImageNet ConViT-Ti Top 1 Accuracy 73.1% # 662
Number of params 6M # 286
GFLOPs 1 # 92