CLIP Surgery for Better Explainability with Enhancement in Open-Vocabulary Tasks

12 Apr 2023  ·  Yi Li, Hualiang Wang, Yiqun Duan, Xiaomeng Li ·

Contrastive Language-Image Pre-training (CLIP) is a powerful multimodal large vision model that has demonstrated significant benefits for downstream tasks, including many zero-shot learning and text-guided vision tasks. However, we notice some severe problems regarding the model's explainability, which undermines its credibility and impedes related tasks. Specifically, we find CLIP prefers the background regions than the foregrounds according to the predicted similarity map, which contradicts human understanding. Besides, there are obvious noisy activations on the visualization results at irrelevant positions. To address these two issues, we conduct in-depth analyses and reveal the reasons with new findings and evidences. Based on these insights, we propose the CLIP Surgery, a method that enables surgery-like modifications for the inference architecture and features, for better explainability and enhancement in multiple open-vocabulary tasks. The proposed method has significantly improved the explainability of CLIP for both convolutional networks and vision transformers, surpassing existing methods by large margins. Besides, our approach also demonstrates remarkable improvements in open-vocabulary segmentation and multi-label recognition tasks. For examples, the mAP improvement on NUS-Wide multi-label recognition is 4.41% without any additional training, and our CLIP Surgery surpasses the state-of-the-art method by 8.74% at mIoU on Cityscapes open-vocabulary semantic segmentation. Furthermore, our method benefits other tasks including multimodal visualization and interactive segmentation like Segment Anything Model (SAM). The code is available at

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero Shot Segmentation ADE20K training-free zero-shot segmentation CLIPSurgery mIoU 12.9 # 3
Open Vocabulary Semantic Segmentation Cityscapes CLIP Surgery (CLIP without any fine-tuning) mIoU 31.4 # 4
Open Vocabulary Semantic Segmentation COCO-Stuff-171 CLIP Surgery (original CLIP without any fine-tuning) mIoU 21.9 # 2
Open Vocabulary Semantic Segmentation PASCAL Context-59 CLIP Surgery (original CLIP without any fine-tuning) mIoU 29.3 # 18