MetaFormer Baselines for Vision

24 Oct 2022  ·  Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang ·

MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, without focusing on token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with GELU yet achieves better performance. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

PDF Abstract

Results from the Paper


Ranked #2 on Domain Generalization on ImageNet-C (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Image Classification ImageNet CAFormer-B36 (384 res) Top 1 Accuracy 86.4% # 142
Number of params 99M # 863
GFLOPs 72.2 # 447
Image Classification ImageNet CAFormer-M36 (224 res, 21K) Top 1 Accuracy 86.6% # 131
Number of params 56M # 747
GFLOPs 13.2 # 327
Image Classification ImageNet CAFormer-S36 (384 res, 21K) Top 1 Accuracy 86.9% # 114
Number of params 39M # 665
GFLOPs 26.0 # 387
Image Classification ImageNet CAFormer-S36 (224 res, 21K) Top 1 Accuracy 85.8% # 186
Number of params 39M # 665
GFLOPs 8.0 # 268
Image Classification ImageNet ConvFormer-S36 (384 res, 21K) Top 1 Accuracy 86.4% # 142
Number of params 40M # 677
GFLOPs 22.4 # 375
Image Classification ImageNet ConvFormer-S36 (224 res, 21K) Top 1 Accuracy 85.4% # 221
Number of params 40M # 677
GFLOPs 7.6 # 259
Image Classification ImageNet CAFormer-S18 (384 res, 21K) Top 1 Accuracy 85.4% # 221
Number of params 26M # 606
GFLOPs 13.4 # 330
Image Classification ImageNet CAFormer-S18 (224 res, 21K) Top 1 Accuracy 84.1% # 328
Number of params 26M # 606
GFLOPs 4.1 # 196
Image Classification ImageNet ConvFormer-S18 (384 res, 21K) Top 1 Accuracy 85.0% # 255
Number of params 27M # 614
GFLOPs 11.6 # 314
Image Classification ImageNet ConvFormer-S18 (224 res, 21K) Top 1 Accuracy 83.7% # 369
Number of params 27M # 614
GFLOPs 3.9 # 189
Image Classification ImageNet CAFormer-S18 (224 res) Top 1 Accuracy 83.6% # 383
Number of params 26M # 606
GFLOPs 4.1 # 196
Image Classification ImageNet ConvFormer-B36 (224 res, 21K) Top 1 Accuracy 87.0% # 111
Number of params 100M # 870
GFLOPs 22.6 # 377
Image Classification ImageNet ConvFormer-B36 (384 res, 21K) Top 1 Accuracy 87.6% # 83
Number of params 100M # 870
GFLOPs 66.5 # 443
Image Classification ImageNet CAFormer-M36 (384 res) Top 1 Accuracy 86.2% # 163
Number of params 56M # 747
GFLOPs 42.0 # 420
Image Classification ImageNet CAFormer-S36 (384 res) Top 1 Accuracy 85.7% # 200
Number of params 39M # 665
GFLOPs 26.0 # 387
Image Classification ImageNet CAFormer-S36 (224 res) Top 1 Accuracy 84.5% # 295
Number of params 39M # 665
GFLOPs 8.0 # 268
Image Classification ImageNet ConvFormer-S18 (224 res) Top 1 Accuracy 83.0% # 442
Number of params 27M # 614
GFLOPs 3.9 # 189
Image Classification ImageNet ConvFormer-S36 (224 res) Top 1 Accuracy 84.1% # 328
Number of params 40M # 677
GFLOPs 7.6 # 259
Image Classification ImageNet ConvFormer-S18 (384 res) Top 1 Accuracy 84.4% # 301
Number of params 27M # 614
GFLOPs 11.6 # 314
Image Classification ImageNet ConvFormer-M36 (224 res) Top 1 Accuracy 84.5% # 295
Number of params 57M # 754
GFLOPs 12.8 # 323
Image Classification ImageNet CAFormer-S18 (384 res) Top 1 Accuracy 85.0% # 255
Number of params 26M # 606
GFLOPs 13.4 # 330
Image Classification ImageNet CAFormer-M36 (224 res) Top 1 Accuracy 85.2% # 239
Number of params 56M # 747
GFLOPs 13.2 # 327
Image Classification ImageNet ConvFormer-S36 (384 res) Top 1 Accuracy 85.4% # 221
Number of params 40M # 677
GFLOPs 22.4 # 375
Image Classification ImageNet ConvFormer-M36 (384 res) Top 1 Accuracy 85.6% # 209
Number of params 57M # 754
GFLOPs 37.7 # 414
Image Classification ImageNet ConvFormer-B36 (224 res) Top 1 Accuracy 84.8% # 270
Number of params 100M # 870
GFLOPs 22.6 # 377
Image Classification ImageNet CAFormer-B36 (224 res) Top 1 Accuracy 85.5% # 212
Number of params 99M # 863
GFLOPs 23.2 # 379
Image Classification ImageNet ConvFormer-B36 (384 res) Top 1 Accuracy 85.7% # 200
Number of params 100M # 870
GFLOPs 66.5 # 443
Image Classification ImageNet CAFormer-B36 (384 res, 21K) Top 1 Accuracy 88.1% # 65
Number of params 99M # 863
GFLOPs 72.2 # 447
Image Classification ImageNet CAFormer-B36 (224 res, 21K) Top 1 Accuracy 87.4% # 92
Number of params 99M # 863
GFLOPs 23.2 # 379
Image Classification ImageNet CAFormer-M36 (384 res, 21K) Top 1 Accuracy 87.5% # 85
Number of params 56M # 747
GFLOPs 42 # 420
Image Classification ImageNet ConvFormer-M36 (384 res, 21K) Top 1 Accuracy 86.9% # 114
Number of params 57M # 754
GFLOPs 37.7 # 414
Image Classification ImageNet ConvFormer-M36 (224 res, 21K) Top 1 Accuracy 86.1% # 169
Number of params 57M # 754
GFLOPs 12.8 # 323
Domain Generalization ImageNet-A ConvFormer-B36 Top-1 accuracy % 40.1 # 23
Number of params 100M # 1
Domain Generalization ImageNet-A ConvFormer-B36 (384) Top-1 accuracy % 55.3 # 17
Number of params 100M # 1
Domain Generalization ImageNet-A CAFormer-B36 (IN-21K) Top-1 accuracy % 69.4 # 9
Number of params 99M # 5
Domain Generalization ImageNet-A CAFormer-B36 (IN-21K, 384) Top-1 accuracy % 79.5 # 5
Number of params 99M # 5
Domain Generalization ImageNet-A ConvFormer-B36 (IN-21K) Top-1 accuracy % 63.3 # 12
Number of params 100M # 1
Domain Generalization ImageNet-A ConvFormer-B36 (IN-21K, 384) Top-1 accuracy % 73.5 # 8
Number of params 100M # 1
Domain Generalization ImageNet-A CAFormer-B36 Top-1 accuracy % 48.5 # 20
Number of params 99M # 5
Domain Generalization ImageNet-A CAFormer-B36 (384) Top-1 accuracy % 61.9 # 14
Number of params 99M # 5
Domain Generalization ImageNet-C ConvFormer-B36 mean Corruption Error (mCE) 46.3 # 23
Domain Generalization ImageNet-C CAFormer-B36 (IN21K, 384) mean Corruption Error (mCE) 30.8 # 2
Number of params 99M # 40
Domain Generalization ImageNet-C CAFormer-B36 mean Corruption Error (mCE) 42.6 # 18
Domain Generalization ImageNet-C ConvFormer-B36 (IN21K) mean Corruption Error (mCE) 35.0 # 7
Domain Generalization ImageNet-C CAFormer-B36 (IN21K) mean Corruption Error (mCE) 31.8 # 5
Domain Generalization ImageNet-R ConvFormer-B36 (IN21K, 384) Top-1 Error Rate 33.5 # 10
Domain Generalization ImageNet-R CAFormer-B36 (IN21K, 384) Top-1 Error Rate 29.6 # 5
Domain Generalization ImageNet-R CAFormer-B36 (IN21K) Top-1 Error Rate 31.7 # 7
Domain Generalization ImageNet-R ConvFormer-B36 Top-1 Error Rate 48.9 # 25
Domain Generalization ImageNet-R ConvFormer-B36 (384) Top-1 Error Rate 47.8 # 24
Domain Generalization ImageNet-R CAFormer-B36 (384) Top-1 Error Rate 45 # 21
Domain Generalization ImageNet-R CAFormer-B36 Top-1 Error Rate 46.1 # 23
Domain Generalization ImageNet-R ConvFormer-B36 (IN21K) Top-1 Error Rate 34.7 # 13
Domain Generalization ImageNet-Sketch CAFormer-B36 (IN21K) Top-1 accuracy 52.8 # 8
Domain Generalization ImageNet-Sketch ConvFormer-B36 (IN21K, 384) Top-1 accuracy 52.9 # 7
Domain Generalization ImageNet-Sketch CAFormer-B36 Top-1 accuracy 42.5 # 17
Domain Generalization ImageNet-Sketch ConvFormer-B36 Top-1 accuracy 39.5 # 19
Domain Generalization ImageNet-Sketch CAFormer-B36 (IN21K, 384) Top-1 accuracy 54.5 # 5
Domain Generalization ImageNet-Sketch ConvFormer-B36 (IN21K) Top-1 accuracy 52.7 # 9

Methods