High-Performance Large-Scale Image Recognition Without Normalization

11 Feb 2021  ·  Andrew Brock, Soham De, Samuel L. Smith, Karen Simonyan ·

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

PDF Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Image Classification ImageNet NFNet-F4+ Top 1 Accuracy 89.2% # 30
Number of params 527M # 941
GFLOPs 367 # 478
Image Classification ImageNet NFNet-F2 Top 1 Accuracy 85.1% # 242
Number of params 193.8M # 894
GFLOPs 62.59 # 434
Image Classification ImageNet NFNet-F4 Top 1 Accuracy 85.9% # 181
Number of params 316.1M # 918
GFLOPs 215.24 # 471
Image Classification ImageNet NFNet-F5 Top 1 Accuracy 86.0% # 174
Number of params 377.2M # 926
GFLOPs 289.76 # 474
Image Classification ImageNet NFNet-F5 w/ SAM Top 1 Accuracy 86.3% # 151
Number of params 377.2M # 926
GFLOPs 289.76 # 474
Image Classification ImageNet NFNet-F3 Top 1 Accuracy 85.7% # 197
Number of params 254.9M # 907
GFLOPs 114.76 # 457
Image Classification ImageNet NFNet-F1 Top 1 Accuracy 84.7% # 278
Number of params 132.6M # 879
GFLOPs 35.54 # 402
Image Classification ImageNet NFNet-F0 Top 1 Accuracy 83.6% # 375
Number of params 71.5M # 789
GFLOPs 12.38 # 317
Image Classification ImageNet NFNet-F6 w/ SAM Top 1 Accuracy 86.5% # 133
Number of params 438.4M # 931
GFLOPs 377.28 # 479

Methods