Large Scale Incremental Learning

Modern machine learning suffers from catastrophic forgetting when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes. Incremental learning methods have been proposed to retain the knowledge acquired from the old classes, by using knowledge distilling and keeping a few exemplars from the old classes. However, these methods struggle to scale up to a large number of classes. We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes. Distinguishing between an increasing number of visually similar classes is particularly challenging, when the training data is unbalanced. We propose a simple and effective method to address this data imbalance issue. We found that the last fully connected layer has a strong bias towards the new classes, and this bias can be corrected by a linear model. With two bias parameters, our method performs remarkably well on two large datasets: ImageNet (1000 classes) and MS-Celeb-1M (10000 classes), outperforming the state-of-the-art algorithms by 11.1% and 13.2% respectively.

PDF Abstract CVPR 2019 PDF CVPR 2019 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Incremental Learning ImageNet100 - 10 steps BiC Average Incremental Accuracy Top-5 90.60 # 5
Final Accuracy Top-5 84.40 # 4
# M Params 11.22 # 2
Incremental Learning ImageNet-100 - 50 classes + 50 steps of 1 class BiC Average Incremental Accuracy 46.49 # 2
Incremental Learning ImageNet - 10 steps BiC Average Incremental Accuracy Top-5 84.00 # 5
Final Accuracy Top-5 73.20 # 5
# M Params 11.68 # 2

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Incremental Learning CIFAR-100 - 50 classes + 10 steps of 5 classes BiC Average Incremental Accuracy 53.21 # 9
Incremental Learning CIFAR-100 - 50 classes + 25 steps of 2 classes BiC Average Incremental Accuracy 48.96 # 4
Incremental Learning CIFAR-100 - 50 classes + 50 steps of 1 class BiC Average Incremental Accuracy 47.09 # 2
Incremental Learning CIFAR-100 - 50 classes + 5 steps of 10 classes BiC Average Incremental Accuracy 56.86 # 10

Methods


No methods listed for this paper. Add relevant methods here