SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Overlapped 50-50 ADE20K SSUL-M mIoU 29.77 # 4
Overlapped 50-50 ADE20K SSUL mIoU 29.56 # 5
Overlapped 100-50 ADE20K SSUL-M mIoU 34.37 # 3
Overlapped 100-50 ADE20K SSUL mIoU 33.58 # 4
Overlapped 100-5 ADE20K SSUL-M mIoU 34.56 # 1
Overlapped 100-5 ADE20K SSUL mIoU 32.48 # 2
Disjoint 15-1 PASCAL VOC 2012 SSUL-M mIoU 68.58 # 1
Disjoint 15-1 PASCAL VOC 2012 SSUL mIoU 64.01 # 2
Overlapped 19-1 PASCAL VOC 2012 SSUL-M Mean IoU (val) 76.49 # 1
Overlapped 19-1 PASCAL VOC 2012 SSUL Mean IoU (val) 75.44 # 2
Overlapped 15-5 PASCAL VOC 2012 SSUL-M Mean IoU (val) 73.02 # 3
Overlapped 15-5 PASCAL VOC 2012 SSUL Mean IoU (val) 71.22 # 6
Overlapped 15-1 PASCAL VOC 2012 SSUL mIoU 67.61 # 4
Disjoint 15-5 PASCAL VOC 2012 SSUL-M Mean IoU 69.83 # 1
Overlapped 10-1 PASCAL VOC 2012 SSUL mIoU 59.25 # 5
Overlapped 15-1 PASCAL VOC 2012 SSUL-M mIoU 71.37 # 3
Overlapped 10-1 PASCAL VOC 2012 SSUL-M mIoU 64.12 # 2
Disjoint 15-5 PASCAL VOC 2012 SSUL Mean IoU 69.10 # 2
Disjoint 10-1 PASCAL VOC 2012 SSUL-M mIoU 53.50 # 1
Disjoint 10-1 PASCAL VOC 2012 SSUL mIoU 50.87 # 2

Methods