RVOS: End-to-End Recurrent Network for Video Object Segmentation

Multiple object video object segmentation is a challenging task, specially for the zero-shot case, when no object mask is given at the initial frame and the model has to find the objects to be segmented along the sequence. In our work, we propose a Recurrent network for multiple object Video Object Segmentation (RVOS) that is fully end-to-end trainable. Our model incorporates recurrence on two different domains: (i) the spatial, which allows to discover the different object instances within a frame, and (ii) the temporal, which allows to keep the coherence of the segmented objects along time. We train RVOS for zero-shot video object segmentation and are the first ones to report quantitative results for DAVIS-2017 and YouTube-VOS benchmarks. Further, we adapt RVOS for one-shot video object segmentation by using the masks obtained in previous time steps as inputs to be processed by the recurrent module. Our model reaches comparable results to state-of-the-art techniques in YouTube-VOS benchmark and outperforms all previous video object segmentation methods not using online learning in the DAVIS-2017 benchmark. Moreover, our model achieves faster inference runtimes than previous methods, reaching 44ms/frame on a P100 GPU.

PDF Abstract CVPR 2019 PDF CVPR 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Unsupervised Video Object Segmentation DAVIS 2017 (test-dev) RVOS J&F 22.5 # 6
Jaccard (Mean) 17.7 # 5
Jaccard (Recall) 16.2 # 5
Jaccard (Decay) 1.6 # 1
F-measure (Mean) 27.3 # 5
F-measure (Recall) 24.8 # 5
F-measure (Decay) 1.8 # 1
Semi-Supervised Video Object Segmentation DAVIS 2017 (test-dev) RVOS J&F 50.3 # 55
Jaccard (Mean) 47.9 # 53
Jaccard (Recall) 54.4 # 13
Jaccard (Decay) 35.7 # 22
F-measure (Mean) 52.6 # 53
F-measure (Recall) 61.7 # 15
F-measure (Decay) 36.7 # 20
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) RVOS Jaccard (Mean) 57.5 # 69
Jaccard (Recall) 65.2 # 21
Jaccard (Decay) 24.9 # 21
F-measure (Mean) 63.6 # 70
F-measure (Recall) 73.2 # 19
F-measure (Decay) 28.2 # 21
J&F 60.55 # 71
Unsupervised Video Object Segmentation DAVIS 2017 (val) RVOS J&F 41.2 # 10
Jaccard (Mean) 36.8 # 10
Jaccard (Recall) 40.2 # 8
F-measure (Mean) 45.7 # 10
F-measure (Recall) 46.4 # 8
One-shot visual object segmentation YouTube-VOS RVOS-Mask-ST+ Jaccard (Seen) 63.6 # 1
Jaccard (Unseen) 45.5 # 1
F-Measure (Seen) 67.2 # 1
F-Measure (Unseen) 51 # 1
Semi-Supervised Video Object Segmentation YouTube-VOS 2018 RVOS F-Measure (Seen) 67.2 # 49
F-Measure (Unseen) 51.0 # 50
Overall 56.8 # 51
Speed (FPS) 45.5 # 4
Jaccard (Seen) 63.6 # 49

Methods


No methods listed for this paper. Add relevant methods here