MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation

We address the problem of semi-supervised video object segmentation (VOS), where the masks of objects of interests are given in the first frame of an input video. To deal with challenging cases where objects are occluded or missing, previous work relies on greedy data association strategies that make decisions for each frame individually. In this paper, we propose a novel approach to defer the decision making for a target object in each frame, until a global view can be established with the entire video being taken into consideration. Our approach is in the same spirit as Multiple Hypotheses Tracking (MHT) methods, making several critical adaptations for the VOS problem. We employ the bounding box (bbox) hypothesis for tracking tree formation, and the multiple hypotheses are spawned by propagating the preceding bbox into the detected bbox proposals within a gated region starting from the initial object mask in the first frame. The gated region is determined by a gating scheme which takes into account a more comprehensive motion model rather than the simple Kalman filtering model in traditional MHT. To further design more customized algorithms tailored for VOS, we develop a novel mask propagation score instead of the appearance similarity score that could be brittle due to large deformations. The mask propagation score, together with the motion score, determines the affinity between the hypotheses during tree pruning. Finally, a novel mask merging strategy is employed to handle mask conflicts between objects. Extensive experiments on challenging datasets demonstrate the effectiveness of the proposed method, especially in the case of object missing.

PDF Abstract CVPR 2019 PDF CVPR 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semi-Supervised Video Object Segmentation DAVIS 2016 MHP-VOS Jaccard (Mean) 87.6 # 38
Jaccard (Recall) 97.3 # 4
Jaccard (Decay) 6.9 # 20
F-measure (Mean) 89.5 # 39
F-measure (Recall) 95.5 # 6
F-measure (Decay) 9.0 # 16
J&F 88.55 # 40
Semi-Supervised Video Object Segmentation DAVIS 2017 (test-dev) MHP-VOS J&F 69.5 # 40
Jaccard (Mean) 66.4 # 40
Jaccard (Recall) 76.0 # 5
Jaccard (Decay) 18.0 # 5
F-measure (Mean) 72.7 # 40
F-measure (Recall) 82.3 # 5
F-measure (Decay) 19.1 # 6
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) MHP-VOS Jaccard (Mean) 73.4 # 50
Jaccard (Recall) 83.5 # 7
Jaccard (Decay) 17.8 # 14
F-measure (Mean) 78.9 # 51
F-measure (Recall) 87.2 # 6
F-measure (Decay) 19.1 # 10
J&F 76.15 # 52

Methods


No methods listed for this paper. Add relevant methods here