TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Image Captioning	COCO Captions	VinVL	BLEU-4	41.0	# 15
Image Captioning	COCO Captions	VinVL	METEOR	31.1	# 10
Image Captioning	COCO Captions	VinVL	CIDER	140.9	# 16
Image Captioning	COCO Captions	VinVL	SPICE	25.2	# 8
Image-text matching	CommercialAdsDataset	VinVL	ADD(S) AUC	88.56	# 2
Visual Question Answering (VQA)	GQA Test2019	Single Model	Accuracy	64.65	# 11
Visual Question Answering (VQA)	GQA Test2019	Single Model	Binary	82.63	# 3
Visual Question Answering (VQA)	GQA Test2019	Single Model	Open	48.77	# 14
Visual Question Answering (VQA)	GQA Test2019	Single Model	Consistency	94.35	# 4
Visual Question Answering (VQA)	GQA Test2019	Single Model	Plausibility	84.98	# 25
Visual Question Answering (VQA)	GQA Test2019	Single Model	Validity	96.62	# 7
Visual Question Answering (VQA)	GQA Test2019	Single Model	Distribution	4.72	# 114
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	92.46	# 13
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	B1	81.59	# 11
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	B2	65.15	# 11
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	B3	45.04	# 13
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	B4	26.15	# 13
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	ROUGE-L	56.96	# 12
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	METEOR	27.57	# 13
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	SPICE	13.07	# 13
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	97.99	# 15
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	B1	83.24	# 11
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	B2	68.04	# 12
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	B3	49.68	# 14
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	B4	30.62	# 14
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	ROUGE-L	58.54	# 14
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	METEOR	29.51	# 13
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	SPICE	13.63	# 14
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	95.16	# 13
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	B1	82.77	# 11
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	B2	66.94	# 11
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	B3	47.02	# 13
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	B4	27.97	# 13
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	ROUGE-L	57.95	# 13
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	METEOR	28.24	# 14
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	SPICE	13.36	# 15
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	78.01	# 15
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	B1	75.78	# 14
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	B2	56.1	# 16
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	B3	34.02	# 16
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	B4	15.86	# 17
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	ROUGE-L	51.99	# 14
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	METEOR	23.55	# 17
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	SPICE	11.48	# 15
Image Captioning	nocaps-val-in-domain	VinVL	CIDEr	103.1	# 10
Image Captioning	nocaps-val-in-domain	VinVL	SPICE	14.2	# 9
Image Captioning	nocaps-val-in-domain	VinVL	Pre-train (#images)	5.7M	# 5
Image Captioning	nocaps-val-near-domain	VinVL	CIDEr	96.1	# 9
Image Captioning	nocaps-val-near-domain	VinVL	SPICE	13.8	# 8
Image Captioning	nocaps-val-near-domain	VinVL	Pre-train (#images)	5.7M	# 6
Image Captioning	nocaps-val-out-domain	VinVL	CIDEr	88.3	# 10
Image Captioning	nocaps-val-out-domain	VinVL	SPICE	12.1	# 8
Image Captioning	nocaps-val-out-domain	VinVL	Pretrain (#images)	5.7M	# 6
Image Captioning	nocaps-val-overall	VinVL	CIDEr	95.5	# 9
Image Captioning	nocaps-val-overall	VinVL	SPICE	13.5	# 8
Image Captioning	nocaps-val-overall	VinVL	Pretrain (#images)	5.7M	# 6
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs.	overall	76.63	# 13
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs.	yes/no	92.04	# 6
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs.	number	61.5	# 5
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs.	other	66.68	# 6
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs., X10 models	overall	77.45	# 12
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs., X10 models	yes/no	92.38	# 5
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs., X10 models	number	62.55	# 4
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs., X10 models	other	67.87	# 5

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-text-matching-on-commercialadsdataset)](https://paperswithcode.com/sota/image-text-matching-on-commercialadsdataset?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-val-near-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-near-domain?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-val-overall)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-overall?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-val-in-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-in-domain?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-val-out-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-out-domain?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/visual-question-answering-on-gqa-test2019)](https://paperswithcode.com/sota/visual-question-answering-on-gqa-test2019?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/visual-question-answering-on-vqa-v2-test-std)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-std?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-entire)](https://paperswithcode.com/sota/image-captioning-on-nocaps-entire?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-near-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-near-domain?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-coco-captions)](https://paperswithcode.com/sota/image-captioning-on-coco-captions?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-in-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-in-domain?p=vinvl-making-visual-representations-matter-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vinvl-making-visual-representations-matter-in/image-captioning-on-nocaps-out-of-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-out-of-domain?p=vinvl-making-visual-representations-matter-in)`

VinVL: Revisiting Visual Representations in Vision-Language Models

CVPR 2021 · Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao ·

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used \emph{bottom-up and top-down} model \cite{anderson2018bottom}, the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

PDF Abstract CVPR 2021 PDF CVPR 2021 Abstract

Code

Add Remove Mark official

pzzhang/VinVL official

354

microsoft/Oscar

1,048

yaolinli/capenrich

JoshuaPlacidi/MS-COCO-Object-Tags

mkhalil1998/EC601_Group_Project

See all 7 implementations

Tasks

Add Remove

Image Captioning

Image-text matching

Object

object-detection

Object Detection

Visual Reasoning

Datasets

MS COCO

Visual Genome

GQA

Visual Question Answering v2.0

COCO Captions

NoCaps

Objects365 CommercialAdsDataset

Results from the Paper

Edit

Ranked #2 on Image-text matching on CommercialAdsDataset

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Image Captioning	COCO Captions	VinVL	BLEU-4	41.0	# 15	Compare
			METEOR	31.1	# 10	Compare
			CIDER	140.9	# 16	Compare
			SPICE	25.2	# 8	Compare
Image-text matching	CommercialAdsDataset	VinVL	ADD(S) AUC	88.56	# 2	Compare
Visual Question Answering (VQA)	GQA Test2019	Single Model	Accuracy	64.65	# 11	Compare
			Binary	82.63	# 3	Compare
			Open	48.77	# 14	Compare
			Consistency	94.35	# 4	Compare
			Plausibility	84.98	# 25	Compare
			Validity	96.62	# 7	Compare
			Distribution	4.72	# 114	Compare
Image Captioning	nocaps entire	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	92.46	# 13	Compare
			B1	81.59	# 11	Compare
			B2	65.15	# 11	Compare
			B3	45.04	# 13	Compare
			B4	26.15	# 13	Compare
			ROUGE-L	56.96	# 12	Compare
			METEOR	27.57	# 13	Compare
			SPICE	13.07	# 13	Compare
Image Captioning	nocaps in-domain	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	97.99	# 15	Compare
			B1	83.24	# 11	Compare
			B2	68.04	# 12	Compare
			B3	49.68	# 14	Compare
			B4	30.62	# 14	Compare
			ROUGE-L	58.54	# 14	Compare
			METEOR	29.51	# 13	Compare
			SPICE	13.63	# 14	Compare
Image Captioning	nocaps near-domain	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	95.16	# 13	Compare
			B1	82.77	# 11	Compare
			B2	66.94	# 11	Compare
			B3	47.02	# 13	Compare
			B4	27.97	# 13	Compare
			ROUGE-L	57.95	# 13	Compare
			METEOR	28.24	# 14	Compare
			SPICE	13.36	# 15	Compare
Image Captioning	nocaps out-of-domain	VinVL (Microsoft Cognitive Services + MSR)	CIDEr	78.01	# 15	Compare
			B1	75.78	# 14	Compare
			B2	56.1	# 16	Compare
			B3	34.02	# 16	Compare
			B4	15.86	# 17	Compare
			ROUGE-L	51.99	# 14	Compare
			METEOR	23.55	# 17	Compare
			SPICE	11.48	# 15	Compare
Image Captioning	nocaps-val-in-domain	VinVL	CIDEr	103.1	# 10	Compare
			SPICE	14.2	# 9	Compare
			Pre-train (#images)	5.7M	# 5	Compare
Image Captioning	nocaps-val-near-domain	VinVL	CIDEr	96.1	# 9	Compare
			SPICE	13.8	# 8	Compare
			Pre-train (#images)	5.7M	# 6	Compare
Image Captioning	nocaps-val-out-domain	VinVL	CIDEr	88.3	# 10	Compare
			SPICE	12.1	# 8	Compare
			Pretrain (#images)	5.7M	# 6	Compare
Image Captioning	nocaps-val-overall	VinVL	CIDEr	95.5	# 9	Compare
			SPICE	13.5	# 8	Compare
			Pretrain (#images)	5.7M	# 6	Compare
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs.	overall	76.63	# 13	Compare
			yes/no	92.04	# 6	Compare
			number	61.5	# 5	Compare
			other	66.68	# 6	Compare
Visual Question Answering (VQA)	VQA v2 test-std	MSR + MS Cog. Svcs., X10 models	overall	77.45	# 12	Compare
			yes/no	92.38	# 5	Compare
			number	62.55	# 4	Compare
			other	67.87	# 5	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

VinVL: Revisiting Visual Representations in Vision-Language Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove