Explicit Visual Prompting for Low-Level Structure Segmentations

CVPR 2023  ·  Weihuang Liu, Xi Shen, Chi-Man Pun, Xiaodong Cun ·

We consider the generic problem of detecting low-level structures in images, which includes segmenting the manipulated parts, identifying out-of-focus pixels, separating shadow regions, and detecting concealed objects. Whereas each such topic has been typically addressed with a domain-specific solution, we show that a unified approach performs well across all of them. We take inspiration from the widely-used pre-training and then prompt tuning protocols in NLP and propose a new visual prompting model, named Explicit Visual Prompting (EVP). Different from the previous visual prompting which is typically a dataset-level implicit embedding, our key insight is to enforce the tunable parameters focusing on the explicit visual content from each individual image, i.e., the features from frozen patch embeddings and the input's high-frequency components. The proposed EVP significantly outperforms other parameter-efficient tuning protocols under the same amount of tunable parameters (5.7% extra trainable parameters of each task). EVP also achieves state-of-the-art performances on diverse low-level structure segmentation tasks compared to task-specific solutions. Our code is available at: https://github.com/NiFangBaAGe/Explicit-Visual-Prompt.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Camouflaged Object Segmentation CAMO EVPv1 MAE 0.059 # 6
Weighted F-Measure 0.777 # 6
S-Measure 0.846 # 5
Camouflaged Object Segmentation COD EVPv1 MAE 0.029 # 4
Weighted F-Measure 0.742 # 6
S-Measure 0.843 # 5
Salient Object Detection DUT-OMRON EVPv1 max_F1 0.858 # 1
MAE 0.046 # 3
E-measure 0.894 # 2
S-measure 0.862 # 1
Salient Object Detection DUTS-TE EVPv1 MAE 0.026 # 1
max_F1 0.923 # 1
E-measure 0.947 # 2
S-measure 0.913 # 5
Salient Object Detection ECSSD EVPv1 MAE 0.027 # 1
max_F1 0.960 # 1
S-measure 0.935 # 1
E-measure 0.957 # 1
Salient Object Detection HKU-IS EVPv1 MAE 0.024 # 2
E-measure 0.961 # 2
max_F1 0.952 # 2
S-measure 0.931 # 2
Salient Object Detection PASCAL-S EVPv1 MAE 0.054 # 3
max_F1 0.872 # 4
S-measure 0.878 # 2
E-measure 0.917 # 1

Methods


No methods listed for this paper. Add relevant methods here