Faster Meta Update Strategy for Noise-Robust Deep Learning

CVPR 2021  ·  Youjiang Xu, Linchao Zhu, Lu Jiang, Yi Yang ·

It has been shown that deep neural networks are prone to overfitting on biased training data. Towards addressing this issue, meta-learning employs a meta model for correcting the training bias... Despite the promising performances, super slow training is currently the bottleneck in the meta learning approaches. In this paper, we introduce a novel Faster Meta Update Strategy (FaMUS) to replace the most expensive step in the meta gradient computation with a faster layer-wise approximation. We empirically find that FaMUS yields not only a reasonably accurate but also a low-variance approximation of the meta gradient. We conduct extensive experiments to verify the proposed method on two tasks. We show our method is able to save two-thirds of the training time while still maintaining the comparable or achieving even better generalization performance. In particular, our method achieves the state-of-the-art performance on both synthetic and realistic noisy labels, and obtains promising performance on long-tailed recognition on standard benchmarks. read more

PDF Abstract CVPR 2021 PDF CVPR 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Classification CIFAR-100, 40% Symmetric Noise MentorMix Percentage correct 71.3 # 2
Image Classification CIFAR-100, 40% Symmetric Noise FaMUS Percentage correct 75.91 # 1
Image Classification CIFAR-100, 60% Symmetric Noise MentorMix Percentage correct 64.6 # 1
Image Classification CIFAR-10, 40% Symmetric Noise MentorMix Percentage correct 94.2 # 2
Image Classification CIFAR-10, 40% Symmetric Noise FaMUS Percentage correct 95.37 # 1
Image Classification CIFAR-10, 60% Symmetric Noise FaMUS Percentage correct 26.42 # 2
Image Classification CIFAR-10, 60% Symmetric Noise MentorMix Percentage correct 91.3 # 1
Image Classification mini WebVision 1.0 FaMUS Top-1 Accuracy 79.4 # 6
Top-5 Accuracy 92.80 # 2
ImageNet Top-1 Accuracy 77 # 4
ImageNet Top-5 Accuracy 92.76 # 3
Image Classification Red MiniImageNet 20% label noise FaMUS Accuracy 51.42 # 1
Image Classification Red MiniImageNet 40% label noise FaMUS Accuracy 48.06 # 1
Image Classification Red MiniImageNet 60% label noise FaMUS Accuracy 45.1 # 1
Image Classification Red MiniImageNet 80% label noise FaMUS Accuracy 35.5 # 1

Methods


No methods listed for this paper. Add relevant methods here