CRIS: CLIP-Driven Referring Image Segmentation

Referring image segmentation aims to segment a referent via a natural linguistic expression.Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmentation framework (CRIS). To transfer the multi-modal knowledge effectively, CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances. The experimental results on three benchmark datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art performance without any post-processing. The code will be released.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Referring Expression Segmentation RefCOCO testA CRIS Overall IoU 73.18 # 1
Referring Expression Segmentation RefCOCO+ testA CRIS Overall IoU 68.08 # 2
Referring Expression Segmentation RefCOCO testB CRIS Overall IoU 66.1 # 2
Referring Expression Segmentation RefCOCO+ test B CRIS Overall IoU 53.68 # 3
Referring Expression Segmentation RefCoCo val CRIS Overall IoU 70.47 # 1
Referring Expression Segmentation RefCOCO+ val CRIS Overall IoU 62.27 # 1