CoAtNet: Marrying Convolution and Attention for All Data Sizes

Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Classification GasHisSDB CoAtNet-1 Accuracy 98.74 # 1
Precision 99.97 # 1
F1-Score 99.38 # 1
Image Classification ImageNet CoAtNet-6 Top 1 Accuracy 90.45% # 10
Number of params 1470M # 960
GFLOPs 1521 # 491
Image Classification ImageNet CoAtNet-7 Top 1 Accuracy 90.88% # 7
Number of params 2440M # 970
GFLOPs 2586 # 492
Image Classification ImageNet CoAtNet-3 @384 Top 1 Accuracy 88.52% # 48
Number of params 168M # 884
GFLOPs 114 # 455