Efficient Regional Memory Network for Video Object Segmentation
Recently, several Space-Time Memory based networks have shown that the object cues (e.g. video frames as well as the segmented object masks) from the past frames are useful for segmenting objects in the current frame. However, these methods exploit the information from the memory by global-to-global matching between the current and past frames, which lead to mismatching to similar objects and high computational complexity. To address these problems, we propose a novel local-to-local matching solution for semi-supervised VOS, namely Regional Memory Network (RMNet). In RMNet, the precise regional memory is constructed by memorizing local regions where the target objects appear in the past frames. For the current query frame, the query regions are tracked and predicted based on the optical flow estimated from the previous frame. The proposed local-to-local matching effectively alleviates the ambiguity of similar objects in both memory and query frames, which allows the information to be passed from the regional memory to the query region efficiently and effectively. Experimental results indicate that the proposed RMNet performs favorably against state-of-the-art methods on the DAVIS and YouTube-VOS datasets.
PDF Abstract CVPR 2021 PDF CVPR 2021 AbstractCode
DAVIS
DAVIS 2017
DAVIS 2016
YouTube-VOS 2018
Referring Expressions for DAVIS 2016 & 2017
Results from the Paper
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Semi-Supervised Video Object Segmentation | DAVIS 2016 | RMNet | Jaccard (Mean) | 88.9 | # 30 | ||
| F-measure (Mean) | 88.7 | # 42 | |||||
| J&F | 88.8 | # 39 | |||||
| Semi-Supervised Video Object Segmentation | DAVIS 2017 (test-dev) | RMNet | J&F | 75.0 | # 33 | ||
| Jaccard (Mean) | 71.9 | # 31 | |||||
| F-measure (Mean) | 78.1 | # 34 | |||||
| Semi-Supervised Video Object Segmentation | DAVIS 2017 (val) | RMNet | Jaccard (Mean) | 81.0 | # 31 | ||
| F-measure (Mean) | 86.0 | # 32 | |||||
| J&F | 83.5 | # 32 | |||||
| Semi-Supervised Video Object Segmentation | DAVIS (no YouTube-VOS training) | RMNet | FPS | 11.9 | # 12 | ||
| D16 val (G) | 81.5 | # 16 | |||||
| D16 val (J) | 80.6 | # 14 | |||||
| D16 val (F) | 82.3 | # 14 | |||||
| D17 val (G) | 75.0 | # 9 | |||||
| D17 val (J) | 72.8 | # 10 | |||||
| D17 val (F) | 77.2 | # 10 | |||||
| Semi-Supervised Video Object Segmentation | YouTube-VOS 2018 | RMNet | F-Measure (Seen) | 85.7 | # 37 | ||
| F-Measure (Unseen) | 82.4 | # 40 | |||||
| Overall | 81.5 | # 36 | |||||
| Jaccard (Seen) | 82.1 | # 30 | |||||
| Jaccard (Unseen) | 75.7 | # 35 |
