Look Before You Match: Instance Understanding Matters in Video Object Segmentation

Exploring dense matching between the current frame and past frames for long-range context modeling, memory-based methods have demonstrated impressive results in video object segmentation (VOS) recently. Nevertheless, due to the lack of instance understanding ability, the above approaches are oftentimes brittle to large appearance variations or viewpoint changes resulted from the movement of objects and cameras. In this paper, we argue that instance understanding matters in VOS, and integrating it with memory-based matching can enjoy the synergy, which is intuitively sensible from the definition of VOS task, \ie, identifying and segmenting object instances within the video. Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank. We employ the well-learned object queries from IS branch to inject instance-specific information into the query key, with which the instance-augmented matching is further performed. In addition, we introduce a multi-path fusion block to effectively combine the memory readout with multi-scale features from the instance segmentation decoder, which incorporates high-resolution instance-aware features to produce final segmentation results. Our method achieves state-of-the-art performance on DAVIS 2016/2017 val (92.6% and 87.1%), DAVIS 2017 test-dev (82.8%), and YouTube-VOS 2018/2019 val (86.3% and 86.3%), outperforming alternative methods by clear margins.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Semi-Supervised Video Object Segmentation DAVIS 2016 ISVOS (BL30K, MS) Jaccard (Mean) 92.5 # 1
F-measure (Mean) 94.2 # 5
J&F 93.4 # 1
Video Object Segmentation DAVIS 2016 ISVOS (BL30K, MS) Jaccard (Mean) 92.5 # 1
F-Score 94.2 # 3
J&F 93.4 # 1
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) ISVOS (BL30K) Jaccard (Mean) 84.5 # 8
F-measure (Mean) 91.9 # 4
J&F 88.2 # 5
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) ISVOS (MS) Jaccard (Mean) 85.8 # 4
Jaccard (Recall) 91.4 # 1
J&F 88.6 # 4
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) ISVOS (BL30K, MS) Jaccard (Mean) 86.7 # 2
F-measure (Mean) 93.0 # 2
J&F 89.8 # 2
Semi-Supervised Video Object Segmentation Long Video Dataset ISVOS J&F 90.0 # 1
J 88.3 # 1
F 91.7 # 1

Methods