The devil is in the labels: Semantic segmentation from sentences

4 Feb 2022  Β·  Wei Yin, Yifan Liu, Chunhua Shen, Anton Van Den Hengel, Baichuan Sun Β·

We propose an approach to semantic segmentation that achieves state-of-the-art supervised performance when applied in a zero-shot setting. It thus achieves results equivalent to those of the supervised methods, on each of the major semantic segmentation datasets, without training on those datasets. This is achieved by replacing each class label with a vector-valued embedding of a short paragraph that describes the class. The generality and simplicity of this approach enables merging multiple datasets from different domains, each with varying class labels and semantics. The resulting merged semantic segmentation dataset of over 2 Million images enables training a model that achieves performance equal to that of state-of-the-art supervised methods on 7 benchmark datasets, despite not using any images therefrom. By fine-tuning the model on standard semantic segmentation datasets, we also achieve a significant improvement over the state-of-the-art supervised segmentation on NYUD-V2 and PASCAL-context at 60% and 65% mIoU, respectively. Based on the closeness of language embeddings, our method can even segment unseen labels. Extensive experiments demonstrate strong generalization to unseen image domains and unseen labels, and that the method enables impressive performance improvements in downstream applications, including depth estimation and instance segmentation.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semantic Segmentation CamVid SIW Mean IoU 83.7 # 1
Instance Segmentation COCO minival SIW mask AP 41.4 # 63
Monocular Depth Estimation KITTI Eigen split SIW absolute relative error 0.14 # 54
Semantic Segmentation KITTI Semantic Segmentation SIW Mean IoU (class) 68.9 # 3
Semantic Segmentation PASCAL Context SIW(Segformer-B5) mIoU 54.2 # 30
Semantic Segmentation PASCAL VOC 2010 test SIW Mean IoU 81.1 # 1
Semantic Segmentation PASCAL VOC 2012 val SIW mIoU 65% # 22
Semantic Segmentation WildDash SIW Mean IoU 69.7 # 1


No methods listed for this paper. Add relevant methods here