High-Performance Large-Scale Image Recognition Without Normalization

11 Feb 2021  ·  Andrew Brock, Soham De, Samuel L. Smith, Karen Simonyan ·

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

PDF Abstract

Datasets


Results from the Paper


Ranked #15 on Image Classification on ImageNet (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Image Classification ImageNet NFNet-F4+ Top 1 Accuracy 89.2% # 15
Number of params 527M # 600
Hardware Burden None # 1
Operations per network pass None # 1
Image Classification ImageNet NFNet-F6 w/ SAM Top 1 Accuracy 86.5% # 67
Top 5 Accuracy 97.9% # 13
Number of params 438.4M # 589
Image Classification ImageNet NFNet-F5 w/ SAM Top 1 Accuracy 86.3% # 77
Number of params 377.2M # 586
Image Classification ImageNet NFNet-F5 Top 1 Accuracy 86.0% # 89
Top 5 Accuracy 97.6% # 20
Image Classification ImageNet NFNet-F4 Top 1 Accuracy 85.9% # 94
Number of params 316.1M # 580
Image Classification ImageNet NFNet-F3 Top 1 Accuracy 85.7% # 104
Top 5 Accuracy 97.5% # 23
Number of params 254.9M # 571
Image Classification ImageNet NFNet-F2 Top 1 Accuracy 85.1% # 131
Top 5 Accuracy 97.3% # 31
Number of params 193.8M # 562
Image Classification ImageNet NFNet-F1 Top 1 Accuracy 84.7% # 151
Top 5 Accuracy 97.1% # 36
Number of params 132.6M # 551
Image Classification ImageNet NFNet-F0 Top 1 Accuracy 83.6% # 206
Top 5 Accuracy 96.8% # 47
Number of params 71.5M # 502

Methods