Unsupervised Semantic Segmentation by Distilling Feature Correspondences

Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters. Unlike previous works which achieve this with a single end-to-end framework, we propose to separate feature learning from cluster compactification. Empirically, we show that current unsupervised feature learning frameworks already generate dense features whose correlations are semantically consistent. This observation motivates us to design STEGO ($\textbf{S}$elf-supervised $\textbf{T}$ransformer with $\textbf{E}$nergy-based $\textbf{G}$raph $\textbf{O}$ptimization), a novel framework that distills unsupervised features into high-quality discrete semantic labels. At the core of STEGO is a novel contrastive loss function that encourages features to form compact clusters while preserving their relationships across the corpora. STEGO yields a significant improvement over the prior state of the art, on both the CocoStuff ($\textbf{+14 mIoU}$) and Cityscapes ($\textbf{+9 mIoU}$) semantic segmentation challenges.

PDF Abstract ICLR 2022 PDF ICLR 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Unsupervised Semantic Segmentation Cityscapes test STEGO mIoU 21.0 # 2
Accuracy 73.2 # 3
Unsupervised Semantic Segmentation COCO-All STEGO mIoU 28.2 # 1
Pixel Accuracy 56.9 # 1
Unsupervised Semantic Segmentation COCO-Stuff STEGO (ViT-B/8) Pixel Accuracy 56.9 # 4
mIoU 28.2 # 1
Unsupervised Semantic Segmentation COCO-Stuff STEGO (ViT-S/8) Pixel Accuracy 48.3 # 7
mIoU 24.5 # 3
Unsupervised Semantic Segmentation COCO-Stuff-27 STEGO Accuracy 56.9 # 3
Unsupervised Semantic Segmentation Potsdam-3 STEGO Accuracy 77.0 # 2