SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation

This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. Recent approaches model the RVOS task as a sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct well-aligned joint space at the video level. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin. Besides, the emphasis on temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code will be available.

PDF Abstract NeurIPS 2023 PDF NeurIPS 2023 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Referring Expression Segmentation A2D Sentences SOC (Video-Swin-T) Precision@0.5 0.79 # 4
Precision@0.9 0.195 # 4
IoU overall 0.747 # 4
IoU mean 0.669 # 4
Precision@0.6 0.756 # 4
Precision@0.7 0.687 # 4
Precision@0.8 0.535 # 4
AP 0.504 # 4
Referring Expression Segmentation A2D Sentences SOC (Video-Swin-B) Precision@0.5 0.851 # 1
Precision@0.9 0.252 # 2
IoU overall 0.807 # 1
IoU mean 0.725 # 1
Precision@0.6 0.827 # 1
Precision@0.7 0.765 # 2
Precision@0.8 0.607 # 2
AP 0.573 # 2
Referring Expression Segmentation J-HMDB SOC (Video-Swin-B) Precision@0.5 0.969 # 2
Precision@0.6 0.914 # 2
Precision@0.7 0.711 # 2
Precision@0.8 0.213 # 2
Precision@0.9 0.001 # 5
AP 0.446 # 2
IoU overall 0.736 # 2
IoU mean 0.723 # 2
Referring Expression Segmentation J-HMDB SOC (Video-Swin-T) Precision@0.5 0.947 # 3
Precision@0.6 0.864 # 3
Precision@0.7 0.627 # 3
Precision@0.8 0.179 # 4
Precision@0.9 0.001 # 5
AP 0.397 # 4
IoU overall 0.707 # 3
IoU mean 0.701 # 3
Referring Video Object Segmentation Refer-YouTube-VOS SOC J&F 66.0 # 4
J 64.1 # 4
F 67.9 # 4
Referring Expression Segmentation Refer-YouTube-VOS (2021 public validation) SOC (Joint training, Video-Swin-B) J&F 67.3±0.5 # 6
J 65.3 # 5
F 69.3 # 5
Referring Expression Segmentation Refer-YouTube-VOS (2021 public validation) SOC (Video-Swin-T) J&F 59.2 # 20
J 57.8 # 19
F 60.5 # 19

Methods


No methods listed for this paper. Add relevant methods here