Efficient Video Object Segmentation via Network Modulation

Video object segmentation targets at segmenting a specific object throughout a video sequence, given only an annotated first frame. Recent deep learning based approaches find it effective by fine-tuning a general-purpose segmentation model on the annotated frame using hundreds of iterations of gradient descent... Despite the high accuracy these methods achieve, the fine-tuning process is inefficient and fail to meet the requirements of real world applications. We propose a novel approach that uses a single forward pass to adapt the segmentation model to the appearance of a specific object. Specifically, a second meta neural network named modulator is learned to manipulate the intermediate layers of the segmentation network given limited visual and spatial information of the target object. The experiments show that our approach is 70times faster than fine-tuning approaches while achieving similar accuracy. read more

PDF Abstract CVPR 2018 PDF CVPR 2018 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semi-Supervised Video Object Segmentation DAVIS 2016 OSMN Jaccard (Mean) 74.0 # 35
Jaccard (Recall) 87.6 # 26
Jaccard (Decay) 9.0 # 16
F-measure (Mean) 72.9 # 34
F-measure (Recall) 84.0 # 27
F-measure (Decay) 10.6 # 24
J&F 73.45 # 34
Semi-Supervised Video Object Segmentation DAVIS 2017 (test-dev) OSMN J&F 41.3 # 29
Jaccard (Mean) 37.7 # 29
Jaccard (Recall) 38.9 # 22
Jaccard (Decay) 19.0 # 7
F-measure (Mean) 44.9 # 27
F-measure (Recall) 47.4 # 21
F-measure (Decay) 17.4 # 4
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) OSMN Jaccard (Mean) 52.5 # 35
Jaccard (Recall) 60.9 # 22
Jaccard (Decay) 21.5 # 19
F-measure (Mean) 57.1 # 35
F-measure (Recall) 66.1 # 22
F-measure (Decay) 24.3 # 17
J&F 54.8 # 35
Video Instance Segmentation YouTube-VIS validation OSMN AP75 29.1 # 16
AR1 28.6 # 13
AR10 33.1 # 13
Semi-Supervised Video Object Segmentation YouTube-VOS OSMN F-Measure (Seen) 60.1 # 17
F-Measure (Unseen) 44.0 # 17
Overall 51.2 # 17
Speed (FPS) 7.14 # 6
Jaccard (Seen) 60.0 # 16
Jaccard (Unseen) 40.6 # 17
Visual Object Tracking YouTube-VOS OSMN Jaccard (Seen) 60.0 # 3
O (Average of Measures) 51.2 # 4
F-Measure (Seen) 60.1 # 3
F-Measure (Unseen) 44.0 # 4
One-shot visual object segmentation YouTube-VOS OSMN F-Measure (Seen) 60.1 # 3
F-Measure (Unseen) 44.0 # 3
Jaccard (Seen) 60.0 # 2
Jaccard (Unseen) 40.6 # 3

Methods


No methods listed for this paper. Add relevant methods here