Weakly-Supervised Semantic Segmentation with Visual Words Learning and Hybrid Pooling

10 Feb 2022  ·  Lixiang Ru, Bo Du, Yibing Zhan, Chen Wu ·

Weakly-Supervised Semantic Segmentation (WSSS) methods with image-level labels generally train a classification network to generate the Class Activation Maps (CAMs) as the initial coarse segmentation labels. However, current WSSS methods still perform far from satisfactorily because their adopted CAMs 1) typically focus on partial discriminative object regions and 2) usually contain useless background regions. These two problems are attributed to the sole image-level supervision and aggregation of global information when training the classification networks. In this work, we propose the visual words learning module and hybrid pooling approach, and incorporate them in the classification network to mitigate the above problems. In the visual words learning module, we counter the first problem by enforcing the classification network to learn fine-grained visual word labels so that more object extents could be discovered. Specifically, the visual words are learned with a codebook, which could be updated via two proposed strategies, i.e. learning-based strategy and memory-bank strategy. The second drawback of CAMs is alleviated with the proposed hybrid pooling, which incorporates the global average and local discriminative information to simultaneously ensure object completeness and reduce background regions. We evaluated our methods on PASCAL VOC 2012 and MS COCO 2014 datasets. Without any extra saliency prior, our method achieved 70.6% and 70.7% mIoU on the $val$ and $test$ set of PASCAL VOC dataset, respectively, and 36.2% mIoU on the $val$ set of MS COCO dataset, which significantly surpassed the performance of state-of-the-art WSSS methods.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Weakly-Supervised Semantic Segmentation COCO 2014 val VWL-M mIoU 36.1 # 31
Weakly-Supervised Semantic Segmentation COCO 2014 val VWL-L mIoU 36.2 # 30
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 test VWL-M Mean IoU 70.4 # 39
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 test VWL-L (EMANet) Mean IoU 71.1 # 26
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 test VWL-L Mean IoU 70.7 # 32
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 val VWL-M Mean IoU 70.6 # 33
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 val VWL-L (EMANet) Mean IoU 70.8 # 29
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 val VWL-L Mean IoU 70.6 # 33

Methods


No methods listed for this paper. Add relevant methods here