Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement

Weakly-supervised instance segmentation (WSIS) has been considered as a more challenging task than weakly-supervised semantic segmentation (WSSS). Compared to WSSS, WSIS requires instance-wise localization, which is difficult to extract from image-level labels. To tackle the problem, most WSIS approaches use off-the-shelf proposal techniques that require pre-training with instance or object level labels, deviating the fundamental definition of the fully-image-level supervised setting. In this paper, we propose a novel approach including two innovative components. First, we propose a semantic knowledge transfer to obtain pseudo instance labels by transferring the knowledge of WSSS to WSIS while eliminating the need for the off-the-shelf proposals. Second, we propose a self-refinement method to refine the pseudo instance labels in a self-supervised scheme and to use the refined labels for training in an online manner. Here, we discover an erroneous phenomenon, semantic drift, that occurred by the missing instances in pseudo instance labels categorized as background class. This semantic drift occurs confusion between background and instance in training and consequently degrades the segmentation performance. We term this problem as semantic drift problem and show that our proposed self-refinement method eliminates the semantic drift problem. The extensive experiments on PASCAL VOC 2012 and MS COCO demonstrate the effectiveness of our approach, and we achieve a considerable performance without off-the-shelf proposal techniques. The code is available at https://github.com/clovaai/BESTIE.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Point-Supervised Instance Segmentation COCO 2017 val BESTIE (proposal-free) AP 17.7 # 1
AP@50 34.0 # 1
AP@75 16.4 # 1
Image-level Supervised Instance Segmentation COCO 2017 val BESTIE (proposal-free) AP 14.3 # 1
AP@50 28.0 # 1
AP@75 13.2 # 1
Image-level Supervised Instance Segmentation COCO test-dev BESTIE (proposal-free) AP 14.4 # 2
AP@50 28.0 # 1
AP@75 13.5 # 2
Point-Supervised Instance Segmentation COCO test-dev BESTIE (proposal-free) AP 17.8 # 1
AP@50 34.1 # 1
AP@75 16.7 # 1
Weakly-supervised instance segmentation PASCAL VOC 2012 val BESTIE (point label, proposal-free) mAP@0.25 66.4 # 2
mAP@0.5 56.1 # 2
mAP@0.75 30.2 # 2
Average Best Overlap - # 4
Image-level Supervised Instance Segmentation PASCAL VOC 2012 val BESTIE (proposal-free) mAP@0.5 51.0 # 1
mAP@0.25 61.2 # 1
mAP@0.7 31.9 # 1
mAP@0.75 26.6 # 2
Weakly-supervised instance segmentation PASCAL VOC 2012 val BESTIE (image-level label, proposal-free) mAP@0.25 61.2 # 3
mAP@0.5 51.0 # 3
mAP@0.75 26.6 # 3
Average Best Overlap - # 4
Point-Supervised Instance Segmentation PASCAL VOC 2012 val BESTIE (proposal-free) mAP@0.5 66.4 # 1
mAP@0.25 56.1 # 1
mAP@0.7 36.5 # 1
mAP@0.75 30.2 # 1

Methods


No methods listed for this paper. Add relevant methods here