VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

28 Oct 2022  ·  Henghui Ding, Chang Liu, Suchen Wang, Xudong Jiang ·

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produces multiple sets of input-specific queries to represent the diverse comprehensions of language expression. To find the best among these diverse comprehensions, so as to generate a better mask, we propose a Query Balance Module to selectively fuse the corresponding responses of the set of queries. Furthermore, to enhance the model's ability in dealing with diverse language expressions, we consider inter-sample learning to explicitly endow the model with knowledge of understanding different language expressions to the same object. We introduce masked contrastive learning to narrow down the features of different expressions for the same target object while distinguishing the features of different objects. The proposed approach is lightweight and achieves new state-of-the-art referring segmentation results consistently on five datasets.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Referring Video Object Segmentation MeViS VLT+TC J&F 35.5 # 2
J 33.6 # 2
F 37.3 # 2
Referring Expression Segmentation RefCOCOg-val VLT (Swin-B) Overall IoU 63.49 # 4
Referring Expression Segmentation RefCOCO testA VLT Overall IoU 75.96 # 5
Referring Expression Segmentation RefCOCO+ testA VLT Overall IoU 68.43 # 5
Referring Expression Segmentation RefCOCO testB VLT Overall IoU 69.60 # 5
Referring Expression Segmentation RefCOCO+ test B VLT Overall IoU 56.92 # 5
Referring Expression Segmentation RefCoCo val VLT Overall IoU 72.96 # 6
Referring Expression Segmentation RefCOCO+ val VLT Overall IoU 63.53 # 5
Referring Video Object Segmentation Refer-YouTube-VOS VLT J&F 63.8 # 2
J 61.9 # 2
F 65.6 # 2
Referring Expression Segmentation Refer-YouTube-VOS (2021 public validation) VLT J&F 63.8 # 8
J 61.9 # 7
F 65.6 # 7