DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensembling framework where instance segmentation and semantic correspondence are jointly guided by a structured teacher in addition to the bounding box supervision. The teacher is a structured energy model incorporating a pairwise potential and a cross-image potential to model the pairwise pixel relationships both within and across the boxes. Minimizing the teacher energy simultaneously yields refined object masks and dense correspondences between intra-class objects, which are taken as pseudo-labels to supervise the task network and provide positive/negative correspondence pairs for dense constrastive learning. We show a symbiotic relationship where the two tasks mutually benefit from each other. Our best model achieves 37.9% AP on COCO instance segmentation, surpassing prior weakly supervised methods and is competitive to supervised methods. We also obtain state of the art weakly supervised results on PASCAL VOC12 and PF-PASCAL with real-time inference.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Weakly-supervised instance segmentation COCO 2017 val DiscoBox (ResNet-50) AP 31.4 # 1
AP@50 52.6 # 1
AP@75 32.2 # 1
AP@S 11.5 # 1
AP@M 33.8 # 1
AP@L 50.1 # 1
Box-supervised Instance Segmentation COCO test-dev DiscoBox mask AP 37.9 # 3
Weakly-supervised instance segmentation COCO test-dev DiscoBox (ResNeXt-101-DCN-FPN) AP 37.9 # 1
AP@50 61.4 # 1
AP@75 40.0 # 1
AP@S 18.0 # 1
AP@M 41.1 # 1
AP@L 53.9 # 1
Weakly-supervised instance segmentation COCO test-dev DiscoBox (ResNet-101-DCN-FPN) AP 35.8 # 2
AP@50 59.8 # 2
AP@75 36.4 # 2
AP@S 16.9 # 3
AP@M 38.7 # 2
AP@L 52.1 # 2
Weakly-supervised instance segmentation COCO test-dev DiscoBox (ResNet-50-FPN) AP 32.0 # 7
AP@50 53.6 # 7
AP@75 32.6 # 6
AP@S 11.7 # 7
AP@M 33.7 # 7
AP@L 48.4 # 4
Box-supervised Instance Segmentation PASCAL VOC 2012 val DiscoBox AP_25 75.2 # 3
AP_50 63.6 # 3
AP_70 45.5 # 2
AP_75 37.5 # 2

Methods


No methods listed for this paper. Add relevant methods here