Video Object Segmentation with Language Referring Expressions

21 Mar 2018  ยท  Anna Khoreva, Anna Rohrbach, Bernt Schiele ยท

Most state-of-the-art semi-supervised video object segmentation methods rely on a pixel-accurate mask of a target object provided for the first frame of a video. However, obtaining a detailed segmentation mask is expensive and time-consuming. In this work we explore an alternative way of identifying a target object, namely by employing language referring expressions. Besides being a more practical and natural way of pointing out a target object, using language specifications can help to avoid drift as well as make the system more robust to complex dynamics and appearance variations. Leveraging recent advances of language grounding models designed for images, we propose an approach to extend them to video data, ensuring temporally coherent predictions. To evaluate our method we augment the popular video object segmentation benchmarks, DAVIS'16 and DAVIS'17 with language descriptions of target objects. We show that our language-supervised approach performs on par with the methods which have access to a pixel-level mask of the target object on DAVIS'16 and is competitive to methods using scribbles on the challenging DAVIS'17 dataset.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Object Segmentation DAVIS 2016 VOSwL (Mask+Language) mIoU 84.5 # 1
Video Object Segmentation DAVIS 2016 VOSwL (Language) mIoU 82.8 # 2
Semi-Supervised Video Object Segmentation DAVIS 2016 VOSwL Jaccard (Mean) 83.1 # 56
Jaccard (Recall) 95.7 # 13
Jaccard (Decay) 6.9 # 20
F-measure (Mean) 84.2 # 53
F-measure (Recall) 93.9 # 10
F-measure (Decay) 8.6 # 19
J&F 83.65 # 54
Video Object Segmentation DAVIS 2017 VOSwL (Mask+Language) mIoU 59 # 1
J&F 62.2 # 3
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) VOSwL Jaccard (Recall) 66.1 # 20
Jaccard (Decay) 22.4 # 19
F-measure (Mean) 63.5 # 71
F-measure (Recall) 70.4 # 21
F-measure (Decay) 24.5 # 17
Referring Expression Segmentation DAVIS 2017 (val) Khoreva et al. J&F 1st frame 39.3 # 13
J&F Full video 37.1 # 5
Semi-Supervised Video Object Segmentation DAVIS 2017 (val) VOSwL (Language) Jaccard (Mean) 58.0 # 67
J&F 60.8 # 70

Methods


No methods listed for this paper. Add relevant methods here