PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

Lifelong learning has attracted much attention, but existing works still struggle to fight catastrophic forgetting and accumulate knowledge over long stretches of incremental learning. In this work, we propose PODNet, a model inspired by representation learning. By carefully balancing the compromise between remembering the old classes and learning new ones, PODNet fights catastrophic forgetting, even over very long runs of small incremental tasks --a setting so far unexplored by current works. PODNet innovates on existing art with an efficient spatial-based distillation-loss applied throughout the model and a representation comprising multiple proxy vectors for each class. We validate those innovations thoroughly, comparing PODNet with three state-of-the-art models on three datasets: CIFAR100, ImageNet100, and ImageNet1000. Our results showcase a significant advantage of PODNet over existing art, with accuracy gains of 12.10, 6.51, and 2.85 percentage points, respectively. Code is available at https://github.com/arthurdouillard/incremental_learning.pytorch

PDF Abstract ECCV 2020 PDF ECCV 2020 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Incremental Learning CIFAR-100 - 50 classes + 10 steps of 5 classes PODNet (CNN) Average Incremental Accuracy 63.19 # 4
Incremental Learning CIFAR-100 - 50 classes + 25 steps of 2 classes PODNet Average Incremental Accuracy 60.72 # 1
Incremental Learning CIFAR-100 - 50 classes + 50 steps of 1 class PODNet Average Incremental Accuracy 57.98 # 1
Incremental Learning CIFAR-100 - 50 classes + 5 steps of 10 classes PODNet (CNN) Average Incremental Accuracy 64.83 # 4
Incremental Learning ImageNet-100 - 50 classes + 10 steps of 5 classes PODNet Average Incremental Accuracy 73.14 # 3
Incremental Learning ImageNet-100 - 50 classes + 25 steps of 2 classes PODNet Average Incremental Accuracy 67.28 # 1
Incremental Learning ImageNet-100 - 50 classes + 50 steps of 1 class PODNet Average Incremental Accuracy 62.08 # 1
Incremental Learning ImageNet-100 - 50 classes + 5 steps of 10 classes PODNet Average Incremental Accuracy 75.82 # 2
Incremental Learning ImageNet - 500 classes + 10 steps of 50 classes PODNet Average Incremental Accuracy 64.13 # 1
Incremental Learning ImageNet - 500 classes + 5 steps of 100 classes PODNet Average Incremental Accuracy 66.95 # 2

Methods


No methods listed for this paper. Add relevant methods here