Measuring Robustness to Natural Distribution Shifts in Image Classification

We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at https://modestyachts.github.io/imagenet-testbed/ .

PDF Abstract NeurIPS 2020 PDF NeurIPS 2020 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Domain Generalization VizWiz-Classification ResNet-50 (IN-C) Accuracy - All Images 38.8 # 41
Accuracy - Corrupted Images 33.6 # 37
Accuracy - Clean Images 42.9 # 43
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_greyscale) Accuracy - All Images 30.2 # 84
Accuracy - Corrupted Images 24.3 # 83
Accuracy - Clean Images 34.3 # 84
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_zoom_blur) Accuracy - All Images 32.7 # 81
Accuracy - Corrupted Images 28.3 # 72
Accuracy - Clean Images 36.6 # 81
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_motion_blur) Accuracy - All Images 35.7 # 67
Accuracy - Corrupted Images 30.2 # 62
Accuracy - Clean Images 39.6 # 70
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_fog_aws) Accuracy - All Images 35.9 # 65
Accuracy - Corrupted Images 30.3 # 59
Accuracy - Clean Images 39.9 # 69
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_frost) Accuracy - All Images 36.1 # 63
Accuracy - Corrupted Images 29.7 # 65
Accuracy - Clean Images 40 # 67
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_gaussian_noise) Accuracy - All Images 36.4 # 61
Accuracy - Corrupted Images 30.2 # 62
Accuracy - Clean Images 40.6 # 62
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_jpeg_compression) Accuracy - All Images 36.5 # 59
Accuracy - Corrupted Images 30.3 # 59
Accuracy - Clean Images 41.3 # 58
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_contrast) Accuracy - All Images 36.5 # 59
Accuracy - Corrupted Images 30.7 # 56
Accuracy - Clean Images 40.9 # 60
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_pixelate) Accuracy - All Images 37.4 # 52
Accuracy - Corrupted Images 30.9 # 53
Accuracy - Clean Images 41.4 # 57
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_saturate) Accuracy - All Images 38.2 # 49
Accuracy - Corrupted Images 32.4 # 42
Accuracy - Clean Images 42.4 # 52
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_spatter) Accuracy - All Images 38.3 # 44
Accuracy - Corrupted Images 31.4 # 49
Accuracy - Clean Images 42.7 # 48
Domain Generalization VizWiz-Classification ResNet-50 (IN-C_brightness) Accuracy - All Images 38.8 # 41
Accuracy - Corrupted Images 32.5 # 40
Accuracy - Clean Images 43.5 # 40

Methods


No methods listed for this paper. Add relevant methods here