Search Results for author:
Found 17 papers, 11 papers with code
ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies
We demonstrate our pipeline and create ProPara-Logy, a dataset of analogies between scientific processes.
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains
REVEAL includes comprehensive labels for the relevance, attribution to evidence passages, and logical correctness of each reasoning step in a language model's answer, across a variety of datasets and state-of-the-art language models.
Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment
While existing image-text alignment models reach high quality binary assessments, they fall short of pinpointing the exact source of misalignment.
VideoCon: Robust Video-Language Alignment via Contrast Captions
Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions.
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment.
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters.
Ranked #14 on
Visual Question Answering (VQA)
on InfiMM-Eval
Read, Look or Listen? What's Needed for Solving a Multimodal Dataset
We propose a two-step method to analyze multimodal datasets, which leverages a small seed of human annotation to map each multimodal instance to the modalities required to process it.
Transferring Visual Attributes from Natural Language to Verified Image Generation
In this paper, we propose to strengthen the consistency property of T2I methods in the presence of natural complex language, which often breaks the limits of T2I methods by including non-visual information, and textual elements that require knowledge for accurate generation.
What You See is What You Read? Improving Text-Image Alignment Evaluation
Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks.
Ranked #11 on
Visual Reasoning
on Winoground
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
q2d: Turning Questions into Dialogs to Teach Models How to Search
One of the exciting capabilities of recent language models for dialog is their ability to independently search for relevant information to ground a given dialog response.
IRFL: Image Recognition of Figurative Language
We release our dataset, benchmark, and code, in hopes of driving the development of models that can better understand figurative language.
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
We introduce WHOOPS!, a new dataset and benchmark for visual commonsense.
Ranked #1 on
Image-to-Text Retrieval
on WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
(using extra training data)
VASR: Visual Analogies of Situation Recognition
We leverage situation recognition annotations and the CLIP model to generate a large set of 500k candidate analogies.
Ranked #1 on
Visual Reasoning
on VASR
WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills.
Ranked #1 on
Common Sense Reasoning
on WinoGAViL
Data Efficient Masked Language Modeling for Vision and Language
We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings, aiming for better fusion of text and image in the learned representation.
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA
Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution.
