LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language ("cross-modal") decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Generalized Referring Expression Segmentation gRefCOCO LAVT gIoU 58.40 # 4
cIoU 57.64 # 3
Referring Expression Segmentation RefCOCOg-test LAVT (Swin-B) Overall IoU 62.09 # 9
Referring Expression Segmentation RefCOCOg-val LAVT Overall IoU 61.24 # 12
Referring Expression Segmentation RefCOCO+ testA LAVT Overall IoU 68.38 # 12
Referring Expression Segmentation RefCOCO+ test B LAVT Overall IoU 55.1 # 13
Referring Expression Segmentation RefCOCO+ val LAVT Overall IoU 62.14 # 16

Methods