Sharpness-Aware Minimization for Efficiently Improving Generalization

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality... Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. We open source our code at \url{}. read more

PDF Abstract ICLR 2021 PDF ICLR 2021 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Fine-Grained Image Classification Birdsnap EffNet-L2 (SAM) Accuracy 90.07% # 1
Image Classification CIFAR-10 PyramidNet (SAM) Percentage correct 98.6 # 24
Image Classification CIFAR-100 PyramidNet (SAM) Percentage correct 89.7 # 24
Image Classification CIFAR-100 EffNet-L2 (SAM) Percentage correct 96.08 # 1
Image Classification Fashion-MNIST Shake-Shake (SAM) Percentage error 3.59 # 2
Accuracy 96.41 # 2
Fine-Grained Image Classification FGVC Aircraft EffNet-L2 (SAM) Top-1 Error Rate 4.82 # 1
Image Classification Flowers-102 EffNet-L2 (SAM) Accuracy 99.65% # 3
Fine-Grained Image Classification Food-101 EffNet-L2 (SAM) Accuracy 96.18 # 1
Image Classification ImageNet EfficientNet-L2-475 (SAM) Top 1 Accuracy 88.61% # 13
Number of params 480M # 15
Hardware Burden None # 1
Operations per network pass None # 1
Image Classification ImageNet ResNet-152 (SAM) Top 1 Accuracy 81.6% # 246
Top 5 Accuracy 95.65 # 86
Fine-Grained Image Classification Oxford-IIIT Pets EffNet-L2 (SAM) Top-1 Error Rate 2.90% # 1
Accuracy 97.10% # 1
Fine-Grained Image Classification Stanford Cars EffNet-L2 (SAM) Accuracy 95.96% # 4
Image Classification SVHN WRN28-10 (SAM) Percentage error 0.99 # 1