| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b0_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b1_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b2_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b3_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b4_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b5_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b6_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_b7_ns |
| SHOW MORE |
| Training Techniques | Noisy Student, FixRes, RMSProp, Weight Decay, Label Smoothing, AutoAugment, RandAugment |
|---|---|
| Architecture | 1x1 Convolution, Average Pooling, Convolution, Dense Connections, Dropout, Inverted Residual Block, Batch Normalization, Squeeze-and-Excitation Block, Swish |
| ID | tf_efficientnet_l2_ns |
| SHOW MORE |
Noisy Student Training is a semi-supervised learning approach. It extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. It has three main steps:
The algorithm is iterated a few times by treating the student as a teacher to relabel the unlabeled data and training a new student.
Noisy Student Training seeks to improve on self-training and distillation in two ways. First, it makes the student larger than, or at least equal to, the teacher so the student can better learn from a larger dataset. Second, it adds noise to the student so the noised student is forced to learn harder from the pseudo labels. To noise the student, it uses input noise such as RandAugment data augmentation, and model noise such as dropout and stochastic depth during training.
To load a pretrained model:
import timm
m = timm.create_model('tf_efficientnet_b0_ns', pretrained=True)
m.eval()
Replace the model name with the variant you want to use, e.g. tf_efficientnet_b0_ns. You can find the IDs in the model summaries at the top of this page.
You can follow the timm recipe scripts for training a new model afresh.
@misc{xie2020selftraining,
title={Self-training with Noisy Student improves ImageNet classification},
author={Qizhe Xie and Minh-Thang Luong and Eduard Hovy and Quoc V. Le},
year={2020},
eprint={1911.04252},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Image Classification on ImageNet
| MODEL | TOP 1 ACCURACY | TOP 5 ACCURACY |
|---|---|---|
| tf_efficientnet_l2_ns | 88.35% | 98.66% |
| tf_efficientnet_b7_ns | 86.83% | 98.08% |
| tf_efficientnet_b6_ns | 86.45% | 97.88% |
| tf_efficientnet_b5_ns | 86.08% | 97.75% |
| tf_efficientnet_b4_ns | 85.15% | 97.47% |
| tf_efficientnet_b3_ns | 84.04% | 96.91% |
| tf_efficientnet_b2_ns | 82.39% | 96.24% |
| tf_efficientnet_b1_ns | 81.39% | 95.74% |
| tf_efficientnet_b0_ns | 78.66% | 94.37% |
