SATS: Self-Attention Transfer for Continual Semantic Segmentation

15 Mar 2022  Β·  Yiqiao Qiu, Yixing Shen, Zhuohao Sun, Yanchong Zheng, Xiaobin Chang, Weishi Zheng, Ruixuan Wang Β·

Continually learning to segment more and more types of image regions is a desired capability for many intelligent systems. However, such continual semantic segmentation suffers from the same catastrophic forgetting issue as in continual classification learning. While multiple knowledge distillation strategies originally for continual classification have been well adapted to continual semantic segmentation, they only consider transferring old knowledge based on the outputs from one or more layers of deep fully convolutional networks. Different from existing solutions, this study proposes to transfer a new type of information relevant to knowledge, i.e. the relationships between elements (Eg. pixels or small local regions) within each image which can capture both within-class and between-class knowledge. The relationship information can be effectively obtained from the self-attention maps in a Transformer-style segmentation model. Considering that pixels belonging to the same class in each image often share similar visual properties, a class-specific region pooling is applied to provide more efficient relationship information for knowledge transfer. Extensive evaluations on multiple public benchmarks support that the proposed self-attention transfer method can further effectively alleviate the catastrophic forgetting issue, and its flexible combination with one or more widely adopted strategies significantly outperforms state-of-the-art solutions.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Overlapped 100-10 ADE20K SATS-M Mean IoU (test) 35.45 # 1
Overlapped 25-25 ADE20K SATS-M Mean IoU (test) 32.56 # 1
Overlapped 5-3 PASCAL VOC 2012 SATS Mean IoU (test) 67.36 # 2
Overlapped 5-3 PASCAL VOC 2012 SATS-M Mean IoU (test) 71.36 # 1
Overlapped 15-5 PASCAL VOC 2012 SATS Mean IoU (val) 75.70 # 2
Overlapped 15-1 PASCAL VOC 2012 SATS mIoU 74.48 # 2
Overlapped 10-1 PASCAL VOC 2012 SATS mIoU 61.6 # 3
Overlapped 10-1 PASCAL VOC 2012 SATS-M mIoU 69.27 # 1
Overlapped 15-1 PASCAL VOC 2012 SATS-M mIoU 76.61 # 1
Overlapped 15-5 PASCAL VOC 2012 SATS-M Mean IoU (val) 78.72 # 1

Methods