Self-training with Noisy Student improves ImageNet classification

We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images... On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. Noisy Student Training extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment to the student so that the student generalizes better than the teacher. Models are available at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet. Code is available at https://github.com/google-research/noisystudent. read more

PDF Abstract CVPR 2020 PDF CVPR 2020 Abstract

Results from the Paper


Ranked #9 on Image Classification on ImageNet ReaL (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Image Classification ImageNet NoisyStudent (EfficientNet-B5) Top 1 Accuracy 86.1% # 43
Top 5 Accuracy 97.8% # 15
Number of params 30M # 164
Image Classification ImageNet NoisyStudent (EfficientNet-B1) Top 1 Accuracy 81.5% # 218
Top 5 Accuracy 95.8% # 73
Number of params 7.8M # 245
Image Classification ImageNet NoisyStudent (EfficientNet-B0) Top 1 Accuracy 78.8% # 317
Top 5 Accuracy 94.5% # 126
Number of params 5.3M # 268
Image Classification ImageNet NoisyStudent (EfficientNet-L2) Top 1 Accuracy 88.4% # 14
Top 5 Accuracy 98.7% # 2
Number of params 480M # 14
Hardware Burden 51800G # 1
Operations per network pass None # 1
Image Classification ImageNet NoisyStudent (EfficientNet-B4) Top 1 Accuracy 85.3% # 69
Top 5 Accuracy 97.5% # 21
Number of params 19M # 214
Image Classification ImageNet NoisyStudent (EfficientNet-B3) Top 1 Accuracy 84.1% # 104
Top 5 Accuracy 96.9% # 38
Number of params 12M # 227
Image Classification ImageNet NoisyStudent (EfficientNet-B7) Top 1 Accuracy 86.9% # 28
Top 5 Accuracy 98.1% # 9
Number of params 66M # 96
Image Classification ImageNet NoisyStudent (EfficientNet-B2) Top 1 Accuracy 82.4% # 177
Top 5 Accuracy 96.3% # 59
Number of params 9.2M # 239
Image Classification ImageNet NoisyStudent (EfficientNet-B6) Top 1 Accuracy 86.4% # 35
Top 5 Accuracy 97.9% # 12
Number of params 43M # 140
Image Classification ImageNet ReaL EfficientNet-L2 Accuracy 90.55% # 9
Params 480M # 42

Methods