Texts

VALSE (VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena)

Introduced by Parcalabescu et al. in VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.

Homepage

Benchmarks

Add a new result Link an existing benchmark

Task	Dataset Variant	Best Model
image-sentence alignment	VALSE	ViLBERT 12-in-1
image-sentence alignment	VALSE existence	ViLBERT 12-in-1
image-sentence alignment	VALSE plurality	ViLBERT 12-in-1
image-sentence alignment	VALSE counting balanced	ViLBERT 12-in-1
image-sentence alignment	VALSE counting small numbers	ViLBERT 12-in-1
image-sentence alignment	VALSE counting adversarial	ViLBERT 12-in-1
image-sentence alignment	VALSE spatial relations	GPT1
image-sentence alignment	VALSE action replacement	CLIP
image-sentence alignment	VALSE actant swap	GPT2
image-sentence alignment	VALSE coreference standard	ViLBERT 12-in-1
image-sentence alignment	VALSE coreference clean	ViLBERT 12-in-1
image-sentence alignment	VALSE foil-it (noun phrases)	CLIP