Equivariant Similarity for Vision-Language Foundation Models

This study explores the concept of equivariance in vision-language foundation models (VLMs), focusing specifically on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes. This allows VLMs to generalize better to nuanced and unseen multimodal compositions. However, modeling equivariance is challenging as the ground truth of semantic change is difficult to collect. For example, given an image-text pair about a dog, it is unclear to what extent the similarity changes when the pixel is changed from dog to cat? To this end, we propose EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs and easily pluggable into existing image-text retrieval fine-tuning. Meanwhile, to further diagnose the equivariance of VLMs, we present a new challenging benchmark EqBen. Compared to the existing evaluation sets, EqBen is the first to focus on "visual-minimal change". Extensive experiments show the lack of equivariance in current VLMs and validate the effectiveness of EqSim. Code is available at https://github.com/Wangt-CN/EqBen.

PDF Abstract ICCV 2023 PDF ICCV 2023 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Reasoning Winoground FIBER (EqSim) Text Score 51.5 # 7
Image Score 32.00 # 20
Group Score 27.5 # 17
Visual Reasoning Winoground METER Text Score 39.25 # 37
Image Score 15.75 # 64
Group Score 12.00 # 60
Visual Reasoning Winoground FIBER Text Score 46.25 # 15
Image Score 25.75 # 32
Group Score 22.25 # 24
Visual Reasoning Winoground FIBER (finetuned, Flickr30k) Text Score 51.25 # 8
Image Score 26.50 # 28
Group Score 23.00 # 22
Visual Reasoning Winoground METER (EqSim) Text Score 45.0 # 17
Image Score 22.75 # 42
Group Score 18.75 # 34
Visual Reasoning Winoground METER (finetuned, Flickr30k) Text Score 43.5 # 22
Image Score 20.75 # 49
Group Score 14.75 # 45

Methods


No methods listed for this paper. Add relevant methods here