Search Results for author:
Found 20 papers, 10 papers with code
Vision-Language Pretraining: Current Trends and the Future
In addition to the larger pretraining datasets, the transformer architecture (Vaswani et al., 2017) and in particular self-attention applied to two modalities are responsible for the impressive performance of the recent pretrained models on downstream tasks (Hendricks et al., 2021).
Improving Text-to-Image Consistency via Automatic Prompt Optimization
In this paper, we address these challenges and introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a large language model (LLM) to improve prompt-image consistency in T2I models.
MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model
To enable LLMs to tackle the task in a zero-shot manner, we introduce MoqaGPT, a straightforward and flexible framework.
Improving Automatic VQA Evaluation Using Large Language Models
Thus, there is a need to develop more robust automatic VQA metrics that serve as a proxy for human judgment.
Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
In summary, our research sheds light on the intricacies of prompting strategies in VLMs for VQA, emphasizing the synergistic use of captions, templates, and pre-processing to enhance model efficacy.
Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
However, the compositional reasoning abilities of existing VLMs remains subpar.
An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics
Furthermore, we found that all metrics are sensitive to variations in the size of image-relevant objects mentioned in the caption, while CLIPScore and PAC-S are also sensitive to the number of mentions of image-relevant objects in the caption.
Measuring Progress in Fine-grained Vision-and-Language Understanding
While pretraining on large-scale image-text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained models lack "fine-grained" understanding, such as the ability to recognise relationships, verbs, and numbers in images.
Ranked #13 on
Visual Reasoning
on Winoground
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting
Large pre-trained models have proved to be remarkable zero- and (prompt-based) few-shot learners in unimodal vision and language tasks.
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA).
Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
In this work, we study the setting in which an agent must learn to generate programs for diverse scenes conditioned on a given symbolic instruction.
Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively).
C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.
Measuring Machine Intelligence Through Visual Question Answering
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence.
Analyzing the Behavior of Visual Question Answering Models
Recently, a number of deep-learning based models have been proposed for the task of Visual Question Answering (VQA).
Visual Storytelling
We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling.
Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes
Our approach produces a diverse set of plausible hypotheses for both semantic segmentation and prepositional phrase attachment resolution that are then jointly reranked to select the most consistent pair.
VQA: Visual Question Answering
Given an image and a natural language question about the image, the task is to provide an accurate natural language answer.
