A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection

9 Mar 2022  ·  Yukun Su, Jingliang Deng, Ruizhou Sun, Guosheng Lin, Qingyao Wu ·

Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. In the computer vision area, many researches focus on co-segmentation (CoS), co-saliency detection (CoSD) and video salient object detection (VSOD) to discover the co-occurrent objects. However, previous approaches design different networks on these similar tasks separately, and they are difficult to apply to each other, which lowers the upper bound of the transferability of deep learning frameworks. Besides, they fail to take full advantage of the cues among inter- and intra-feature within a group of images. In this paper, we introduce a unified framework to tackle these issues, term as UFO (Unified Framework for Co-Object Segmentation). Specifically, we first introduce a transformer block, which views the image feature as a patch token and then captures their long-range dependencies through the self-attention mechanism. This can help the network to excavate the patch structured similarities among the relevant objects. Furthermore, we propose an intra-MLP learning module to produce self-mask to enhance the network to avoid partial activation. Extensive experiments on four CoS benchmarks (PASCAL, iCoseg, Internet and MSRC), three CoSD benchmarks (Cosal2015, CoSOD3k, and CocA) and four VSOD benchmarks (DAVIS16, FBMS, ViSal and SegV2) show that our method outperforms other state-of-the-arts on three different tasks in both accuracy and speed by using the same network architecture , which can reach 140 FPS in real-time.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Co-Salient Object Detection CoCA UFO S-measure 0.697 # 4
max F-measure 0.571 # 4
mean E-measure 0.762 # 3
Mean F-measure 0.555 # 4
max E-measure 0.782 # 4
MAE 0.095 # 3
Co-Salient Object Detection CoSal2015 UFO MAE 0.064 # 3
S-measure 0.860 # 4
max F-measure 0.865 # 3
max E-measure 0.906 # 3
mean E-measure 0.889 # 3
mean F-measure 0.848 # 4
Co-Salient Object Detection CoSOD3k UFO S-measure 0.819 # 3
max E-measure 0.874 # 3
max F-measure 0.797 # 4
MAE 0.073 # 5
mean E-measure 0.855 # 5
mean F-measure 0.783 # 4
Video Salient Object Detection DAVIS-2016 UFO S-Measure 0.918 # 1
MAX F-MEASURE 0.906 # 1
AVERAGE MAE 0.015 # 10
Video Salient Object Detection FBMS-59 UFO S-Measure 0.894 # 1
AVERAGE MAE 0.028 # 1
MAX F-MEASURE 0.890 # 1
Co-Salient Object Detection iCoSeg UFO S-measure 0.924 # 1
MAE 0.029 # 1
max E-measure 0.969 # 1
max F-measure 0.953 # 1
Video Salient Object Detection SegTrack v2 UFO S-Measure 0.892 # 1
MAX F-MEASURE 0.863 # 1
AVERAGE MAE 0.022 # 1
Video Salient Object Detection ViSal UFO S-Measure 0.953 # 1
max E-measure 0.987 # 1
Average MAE 0.011 # 1

Methods