Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

16 Nov 2021  ·  Yan Zeng, Xinsong Zhang, Hang Li ·

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Cross-Modal Retrieval COCO 2014 X-VLM (base) Image-to-text R@1 81.2 # 9
Image-to-text R@10 98.2 # 5
Image-to-text R@5 95.6 # 7
Text-to-image R@1 63.4 # 10
Text-to-image R@10 91.5 # 7
Text-to-image R@5 85.8 # 9
Image Captioning COCO Captions X-VLM (base) BLEU-4 41.3 # 13
CIDER 140.8 # 17
Cross-Modal Retrieval Flickr30k X-VLM (base) Image-to-text R@1 97.1 # 6
Image-to-text R@10 100.0 # 1
Image-to-text R@5 100.0 # 1
Text-to-image R@1 86.9 # 7
Text-to-image R@10 98.7 # 8
Text-to-image R@5 97.3 # 9
Image Retrieval Flickr30K 1K test X-VLM (base) R@1 86.9 # 1
R@10 98.7 # 1
R@5 97.3 # 1
Visual Reasoning NLVR2 Dev X-VLM (base) Accuracy 84.41 # 8
Visual Reasoning NLVR2 Test X-VLM (base) Accuracy 84.76 # 8
Open Vocabulary Attribute Detection OVAD-Box benchmark X-VLM mean average precision 28.0 # 1
Visual Grounding RefCOCO+ testA X-VLM (base) Accuracy (%) 89.00 # 5
Visual Grounding RefCOCO+ test B X-VLM (base) Accuracy (%) 76.91 # 5
Visual Grounding RefCOCO+ val X-VLM (base) Accuracy (%) 84.51 # 5
Visual Question Answering (VQA) VQA v2 test-dev X-VLM (base) Accuracy 78.22 # 17

Methods


No methods listed for this paper. Add relevant methods here