Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

31 Oct 2022  ยท  Simone Rossetti, Damiano Zappia, Marta Sanzari, Marco Schaerf, Fiora Pirri ยท

Weakly Supervised Semantic Segmentation (WSSS) research has explored many directions to improve the typical pipeline CNN plus class activation maps (CAM) plus refinements, given the image-class label as the only supervision. Though the gap with the fully supervised methods is reduced, further abating the spread seems unlikely within this framework. On the other hand, WSSS methods based on Vision Transformers (ViT) have not yet explored valid alternatives to CAM. ViT features have been shown to retain a scene layout, and object boundaries in self-supervised learning. To confirm these findings, we prove that the advantages of transformers in self-supervised methods are further strengthened by Global Max Pooling (GMP), which can leverage patch features to negotiate pixel-label probability with class probability. This work proposes a new WSSS method dubbed ViT-PCM (ViT Patch-Class Mapping), not based on CAM. The end-to-end presented network learns with a single optimization process, refined shape and proper localization for segmentation masks. Our model outperforms the state-of-the-art on baseline pseudo-masks (BPM), where we achieve $69.3\%$ mIoU on PascalVOC 2012 $val$ set. We show that our approach has the least set of parameters, though obtaining higher accuracy than all other approaches. In a sentence, quantitative and qualitative results of our method reveal that ViT-PCM is an excellent alternative to CNN-CAM based architectures.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Weakly-Supervised Semantic Segmentation COCO 2014 val ViT-PCM mIoU 45.0 # 14
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 test ViT-PCM Mean IoU 70.9 # 27
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 train ViT-PCM Mean IoU 71.4 # 3
Weakly-Supervised Object Segmentation PASCAL VOC 2012 val ViT-PCM Mean IoU 77.25 # 1
Weakly-Supervised Semantic Segmentation PASCAL VOC 2012 val ViT-PCM Mean IoU 70.3 # 37

Methods