Search Results for author:
Found 24 papers, 10 papers with code
A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models.
Multi-Modal Language Models as Text-to-Image Model Evaluators
The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate the T2I progress.
BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop
We also call for papers outside the competition in any relevant areas.
What makes a good metric? Evaluating automatic metrics for text-to-image consistency
We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information.
EvalGIM: A Library for Evaluating Generative Image Models
To enable actionable evaluation insights, we introduce ''Evaluation Exercises'' that highlight takeaways for specific evaluation questions.
Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
No submissions outperformed the baselines in the multimodal track.
Improving Model Evaluation using SMART Filtering of Benchmark Datasets
To address these concerns, we propose Selection Methodology for Accurate, Reduced, and Targeted (SMART) filtering, a novel approach to select a high-quality subset of examples from existing benchmark datasets by systematically removing less informative and less challenging examples.
Changing Answer Order Can Decrease MMLU Accuracy
As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities.
Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidance
With the growing popularity of text-to-image generative models, there has been increasing focus on understanding their risks and biases.
An Introduction to Vision-Language Modeling
Then, we present and discuss approaches to evaluate VLMs.
Towards Geographic Inclusion in the Evaluation of Text-to-Image Models
We contrast human annotations with common automated metrics, finding that human preferences vary notably across geographic location and that current metrics do not fully account for this diversity.
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
Improving Text-to-Image Consistency via Automatic Prompt Optimization
In this paper, we address these challenges and introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a large language model (LLM) to improve prompt-image consistency in T2I models.
Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision
In this work, we demonstrate that diffusion models can be leveraged to create such a dataset.
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs.
Ranked #2 on
Text-to-Image Generation
on COCO
FACET: Fairness in Computer Vision Evaluation Benchmark
We present a new benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation.
DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity
The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases.
Towards Reliable Assessments of Demographic Disparities in Multi-Label Image Classifiers
These metrics successfully incentivized performance improvements on person-centric tasks such as face analysis and are used to understand risks of modern models.
Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities
With these capabilities in mind, we ask: Do vision-language models exhibit gender bias when performing zero-shot image classification, object detection and semantic segmentation?
Perturbation Augmentation for Fairer NLP
Unwanted and often harmful social biases are becoming ever more salient in NLP research, affecting both models and datasets.
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground.
Ranked #42 on
Visual Reasoning
on Winoground
CM3: A Causal Masked Multimodal Model of the Internet
We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens.
Learning a natural-language to LTL executable semantic parser for grounded robotics
We take a step toward robots that can do the same by training a grounded semantic parser, which discovers latent linguistic representations that can be used for the execution of natural-language commands.
Measuring Social Biases in Grounded Vision and Language Embeddings
We generalize the notion of social biases from language embeddings to grounded vision and language embeddings.
