HTML: Hybrid Temporal-scale Multimodal Learning Framework for Referring Video Object Segmentation
Referring Video Object Segmentation (RVOS) is to segment the object instance from a given video, according to the textual description of this object. However, in the open world, the object descriptions are often diversified in contents and flexible in lengths. This leads to the key difficulty in RVOS, i.e., various descriptions of different ob- jects are corresponding to different temporal scales in the video, which is ignored by most existing approaches with single stride of frame sampling. To tackle this problem, we propose a concise Hybrid Temporal-scale Multimodal Learning (HTML) framework, which can effectively align lingual and visual features to discover core object semantics in the video, by learning multimodal interaction hierarchically from different temporal scales. More specifically, we introduce a novel inter-scale multimodal perception module, where the language queries dynamically interact with visual features across temporal scales. It can effectively reduce complex object confusion by passing video context among different scales. Finally, we conduct extensive experiments on the widely used benchmarks, including Ref- Youtube-VOS, Ref-DAVIS17, A2D-Sentences and JHMDB- Sentences, where our HTML achieves state-of-the-art performance on all these datasets.
PDF Abstract
Datasets
DAVIS
Refer-YouTube-VOS
Results from the Paper
Ranked #6 on
Referring Video Object Segmentation
on Refer-YouTube-VOS
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Referring Video Object Segmentation | Refer-YouTube-VOS | HTML-Video-SwinB | J&F | 63.4 | # 6 | ||
| J | 61.5 | # 6 | |||||
| F | 65.2 | # 7 | |||||
| Referring Video Object Segmentation | Refer-YouTube-VOS | HTML-Video-SwinS | J&F | 61.4 | # 9 | ||
| J | 59.9 | # 9 | |||||
| F | 62.9 | # 10 | |||||
| Referring Video Object Segmentation | Refer-YouTube-VOS | HTML-Video-SwinT | J&F | 61.2 | # 10 | ||
| J | 59.5 | # 10 | |||||
| F | 63.0 | # 9 | |||||
| Referring Video Object Segmentation | Refer-YouTube-VOS | HTML-SwinL | J&F | 63.4 | # 6 | ||
| J | 61.5 | # 6 | |||||
| F | 65.3 | # 6 | |||||
| Referring Video Object Segmentation | Refer-YouTube-VOS | HTML-ResNet101 | J&F | 58.5 | # 12 | ||
| J | 57.3 | # 12 | |||||
| F | 59.8 | # 12 | |||||
| Referring Video Object Segmentation | Refer-YouTube-VOS | HTML-ResNet50 | J&F | 57.8 | # 13 | ||
| J | 56.5 | # 13 | |||||
| F | 59.0 | # 13 |
