TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Video Retrieval	ActivityNet	VALOR	text-to-video R@1	70.1	# 3
Video Retrieval	ActivityNet	VALOR	text-to-video R@5	90.8	# 2
Video Retrieval	ActivityNet	VALOR	text-to-video R@10	95.3	# 2
Video Question Answering	ActivityNet-QA	VALOR	Accuracy	48.6	# 9
Text to Audio Retrieval	AudioCaps	VALOR	R@1	40.1	# 4
Text to Audio Retrieval	AudioCaps	VALOR	R@10	83.1	# 4
Text to Audio Retrieval	AudioCaps	VALOR	R@5	73.9	# 3
Audio captioning	AudioCaps	VALOR	CIDEr	0.741	# 9
Audio captioning	AudioCaps	VALOR	BLEU-4	0.270	# 3
Audio captioning	AudioCaps	VALOR	METEOR	0.231	# 6
Audio captioning	AudioCaps	VALOR	ROUGE-L	0.494	# 3
Zero-shot Text to Audio Retrieval	Clotho	VALOR	text-to-audio R@1	8.4	# 5
Text to Audio Retrieval	Clotho	VALOR	R@1	17.5	# 4
Text to Audio Retrieval	Clotho	VALOR	R@10	55.3	# 3
Text to Audio Retrieval	Clotho	VALOR	R@5	42.7	# 3
Audio captioning	Clotho	VALOR	CIDEr	0.423	# 4
Audio captioning	Clotho	VALOR	BLEU-4	16.2	# 2
Audio captioning	Clotho	VALOR	METEOR	17.4	# 2
Audio captioning	Clotho	VALOR	ROUGE-L	38.2	# 2
Cross-Modal Retrieval	COCO 2014	VALOR	Text-to-image R@1	61.4	# 14
Cross-Modal Retrieval	COCO 2014	VALOR	Text-to-image R@10	90.9	# 11
Cross-Modal Retrieval	COCO 2014	VALOR	Text-to-image R@5	84.4	# 13
Image Captioning	COCO Captions	VALOR	CIDER	152.5	# 3
Image Captioning	COCO Captions	VALOR	SPICE	25.7	# 5
Video Retrieval	DiDeMo	VALOR	text-to-video R@1	61.5	# 6
Video Retrieval	DiDeMo	VALOR	text-to-video R@5	85.3	# 6
Video Retrieval	DiDeMo	VALOR	text-to-video R@10	90.4	# 6
Video Retrieval	LSMDC	VALOR	text-to-video R@1	34.2	# 7
Video Retrieval	LSMDC	VALOR	text-to-video R@5	56.0	# 4
Video Retrieval	LSMDC	VALOR	text-to-video R@10	64.1	# 5
Video Retrieval	MSR-VTT	VALOR	text-to-video R@1	59.9	# 3
Video Retrieval	MSR-VTT	VALOR	text-to-video R@5	83.5	# 2
Video Retrieval	MSR-VTT	VALOR	text-to-video R@10	89.6	# 1
Video Captioning	MSR-VTT	VALOR	CIDEr	74.0	# 6
Video Captioning	MSR-VTT	VALOR	METEOR	32.9	# 5
Video Captioning	MSR-VTT	VALOR	ROUGE-L	68.0	# 4
Video Captioning	MSR-VTT	VALOR	BLEU-4	54.4	# 5
Video Question Answering	MSRVTT-QA	VALOR	Accuracy	49.2	# 3
Video Captioning	MSVD	VALOR	CIDEr	178.5	# 3
Video Captioning	MSVD	VALOR	BLEU-4	80.7	# 1
Video Captioning	MSVD	VALOR	METEOR	51.0	# 2
Video Captioning	MSVD	VALOR	ROUGE-L	87.9	# 1
Visual Question Answering (VQA)	MSVD-QA	VALOR	Accuracy	0.60	# 4
Audio-visual Question Answering	MUSIC-AVQA	VALOR	Acc	78.9	# 2
TGIF-Frame	TGIF-QA	VALOR	Accuracy	78.7	# 4
Audio-Visual Captioning	VALOR-32K	VALOR	METEOR	15.4	# 1
Audio-Visual Captioning	VALOR-32K	VALOR	ROUGE-L	31.8	# 1
Audio-Visual Captioning	VALOR-32K	VALOR	CIDEr	61.5	# 2
Audio-Visual Captioning	VALOR-32K	VALOR	BLEU-4	9.6	# 2
text-to-audiovisual retrieval	VALOR-32K	VALOR	text-to-audiovisual R@1	80.9	# 1
text-to-audiovisual retrieval	VALOR-32K	VALOR	text-to-audiovisual R@5	93.9	# 1
text-to-audiovisual retrieval	VALOR-32K	VALOR	text-to-audiovisual R@10	97.1	# 1
Video Captioning	VATEX	VALOR	BLEU-4	45.6	# 1
Video Captioning	VATEX	VALOR	CIDEr	95.8	# 3
Video Captioning	VATEX	VALOR	METEOR	29.4	# 1
Video Captioning	VATEX	VALOR	ROUGE-L	57.4	# 1
Video Retrieval	VATEX	VALOR	text-to-video R@1	78.5	# 2
Video Retrieval	VATEX	VALOR	text-to-video R@10	98.7	# 2
Video Retrieval	VATEX	VALOR	text-to-video R@5	97.1	# 4
Visual Question Answering (VQA)	VQA v2 test-dev	VALOR	Accuracy	78.46	# 15
Visual Question Answering (VQA)	VQA v2 test-std	VALOR	overall	78.62	# 10

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/text-to-audiovisual-retrieval-on-valor-32k)](https://paperswithcode.com/sota/text-to-audiovisual-retrieval-on-valor-32k?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-captioning-on-vatex-1)](https://paperswithcode.com/sota/video-captioning-on-vatex-1?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/audio-captioning-on-clotho)](https://paperswithcode.com/sota/audio-captioning-on-clotho?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/audio-visual-question-answering-on-music-avqa)](https://paperswithcode.com/sota/audio-visual-question-answering-on-music-avqa?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/audio-visual-captioning-on-valor-32k)](https://paperswithcode.com/sota/audio-visual-captioning-on-valor-32k?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-retrieval-on-vatex)](https://paperswithcode.com/sota/video-retrieval-on-vatex?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-retrieval-on-activitynet)](https://paperswithcode.com/sota/video-retrieval-on-activitynet?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/audio-captioning-on-audiocaps)](https://paperswithcode.com/sota/audio-captioning-on-audiocaps?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/image-captioning-on-coco-captions)](https://paperswithcode.com/sota/image-captioning-on-coco-captions?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-retrieval-on-msr-vtt)](https://paperswithcode.com/sota/video-retrieval-on-msr-vtt?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-question-answering-on-msrvtt-qa)](https://paperswithcode.com/sota/video-question-answering-on-msrvtt-qa?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-captioning-on-msvd-1)](https://paperswithcode.com/sota/video-captioning-on-msvd-1?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/text-to-audio-retrieval-on-audiocaps)](https://paperswithcode.com/sota/text-to-audio-retrieval-on-audiocaps?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/text-to-audio-retrieval-on-clotho)](https://paperswithcode.com/sota/text-to-audio-retrieval-on-clotho?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/visual-question-answering-on-msvd-qa-1)](https://paperswithcode.com/sota/visual-question-answering-on-msvd-qa-1?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/tgif-frame-on-tgif-qa)](https://paperswithcode.com/sota/tgif-frame-on-tgif-qa?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/zero-shot-text-to-audio-retrieval-on-clotho)](https://paperswithcode.com/sota/zero-shot-text-to-audio-retrieval-on-clotho?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-retrieval-on-didemo)](https://paperswithcode.com/sota/video-retrieval-on-didemo?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-captioning-on-msr-vtt-1)](https://paperswithcode.com/sota/video-captioning-on-msr-vtt-1?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-retrieval-on-lsmdc)](https://paperswithcode.com/sota/video-retrieval-on-lsmdc?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/video-question-answering-on-activitynet-qa)](https://paperswithcode.com/sota/video-question-answering-on-activitynet-qa?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/visual-question-answering-on-vqa-v2-test-std)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-std?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/cross-modal-retrieval-on-coco-2014)](https://paperswithcode.com/sota/cross-modal-retrieval-on-coco-2014?p=valor-vision-audio-language-omni-perception)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/valor-vision-audio-language-omni-perception/visual-question-answering-on-vqa-v2-test-dev)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-dev?p=valor-vision-audio-language-omni-perception)`

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

17 Apr 2023 · Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, Jing Liu ·

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

PDF Abstract

Code

Add Remove Mark official

TXH-mercury/VALOR official

232

Tasks

Add Remove

Audio captioning

Audio-Video Question Answering (AVQA)

Audio-Visual Captioning

Audio-visual Question Answering

Audio-Visual Question Answering (AVQA)

Conditional Text Generation

Cross-Modal Retrieval

Image Captioning

Question Answering

Retrieval

Text Generation

Text to Audio Retrieval

text-to-audiovisual retrieval

TGIF-Frame

Video Captioning

Video Question Answering

Video Retrieval

Visual Question Answering (VQA)

Zero-shot Text to Audio Retrieval

Datasets

MS COCO

ActivityNet

AudioSet

MSR-VTT

Visual Question Answering v2.0

MSVD

HowTo100M

DiDeMo

COCO Captions

WebVid

AudioCaps

Clotho

LSMDC

CC12M

VATEX

ActivityNet-QA

TGIF-QA MSRVTT-QA MSVD-QA

VALUE

MUSIC-AVQA

VideoCC3M

Results from the Paper

Edit

Ranked #1 on Video Captioning on VATEX (using extra training data)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Video Retrieval	ActivityNet	VALOR	text-to-video R@1	70.1	# 3	Compare
			text-to-video R@5	90.8	# 2	Compare
			text-to-video R@10	95.3	# 2	Compare
Video Question Answering	ActivityNet-QA	VALOR	Accuracy	48.6	# 9	Compare
Text to Audio Retrieval	AudioCaps	VALOR	R@1	40.1	# 4	Compare
			R@10	83.1	# 4	Compare
			R@5	73.9	# 3	Compare
Audio captioning	AudioCaps	VALOR	CIDEr	0.741	# 9	Compare
			BLEU-4	0.270	# 3	Compare
			METEOR	0.231	# 6	Compare
			ROUGE-L	0.494	# 3	Compare
Zero-shot Text to Audio Retrieval	Clotho	VALOR	text-to-audio R@1	8.4	# 5	Compare
Text to Audio Retrieval	Clotho	VALOR	R@1	17.5	# 4	Compare
			R@10	55.3	# 3	Compare
			R@5	42.7	# 3	Compare
Audio captioning	Clotho	VALOR	CIDEr	0.423	# 4	Compare
			BLEU-4	16.2	# 2	Compare
			METEOR	17.4	# 2	Compare
			ROUGE-L	38.2	# 2	Compare
Cross-Modal Retrieval	COCO 2014	VALOR	Text-to-image R@1	61.4	# 14	Compare
			Text-to-image R@10	90.9	# 11	Compare
			Text-to-image R@5	84.4	# 13	Compare
Image Captioning	COCO Captions	VALOR	CIDER	152.5	# 3	Compare
Image Captioning	COCO Captions	VALOR	SPICE	25.7	# 5	Compare
Video Retrieval	DiDeMo	VALOR	text-to-video R@1	61.5	# 6	Compare
			text-to-video R@5	85.3	# 6	Compare
			text-to-video R@10	90.4	# 6	Compare
Video Retrieval	LSMDC	VALOR	text-to-video R@1	34.2	# 7	Compare
			text-to-video R@5	56.0	# 4	Compare
			text-to-video R@10	64.1	# 5	Compare
Video Retrieval	MSR-VTT	VALOR	text-to-video R@1	59.9	# 3	Compare
			text-to-video R@5	83.5	# 2	Compare
			text-to-video R@10	89.6	# 1	Compare
Video Captioning	MSR-VTT	VALOR	CIDEr	74.0	# 6	Compare
			METEOR	32.9	# 5	Compare
			ROUGE-L	68.0	# 4	Compare
			BLEU-4	54.4	# 5	Compare
Video Question Answering	MSRVTT-QA	VALOR	Accuracy	49.2	# 3	Compare
Video Captioning	MSVD	VALOR	CIDEr	178.5	# 3	Compare
			BLEU-4	80.7	# 1	Compare
			METEOR	51.0	# 2	Compare
			ROUGE-L	87.9	# 1	Compare
Visual Question Answering (VQA)	MSVD-QA	VALOR	Accuracy	0.60	# 4	Compare
Audio-visual Question Answering	MUSIC-AVQA	VALOR	Acc	78.9	# 2	Compare
TGIF-Frame	TGIF-QA	VALOR	Accuracy	78.7	# 4	Compare
Audio-Visual Captioning	VALOR-32K	VALOR	METEOR	15.4	# 1	Compare
			ROUGE-L	31.8	# 1	Compare
			CIDEr	61.5	# 2	Compare
			BLEU-4	9.6	# 2	Compare
text-to-audiovisual retrieval	VALOR-32K	VALOR	text-to-audiovisual R@1	80.9	# 1	Compare
			text-to-audiovisual R@5	93.9	# 1	Compare
			text-to-audiovisual R@10	97.1	# 1	Compare
Video Captioning	VATEX	VALOR	BLEU-4	45.6	# 1	Compare
			CIDEr	95.8	# 3	Compare
			METEOR	29.4	# 1	Compare
			ROUGE-L	57.4	# 1	Compare
Video Retrieval	VATEX	VALOR	text-to-video R@1	78.5	# 2	Compare
			text-to-video R@10	98.7	# 2	Compare
			text-to-video R@5	97.1	# 4	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	VALOR	Accuracy	78.46	# 15	Compare
Visual Question Answering (VQA)	VQA v2 test-std	VALOR	overall	78.62	# 10	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove