Search Results for author:
Found 18 papers, 7 papers with code
Bridging the Gap between Recognition-level Pre-training and Commonsensical Vision-language Tasks
Large-scale visual-linguistic pre-training aims to capture the generic representations from multimodal features, which are essential for downstream vision-language tasks.
JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors.
HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
Our results discover that benchmarking with generated images is highly correlated (r=0. 97) with real images.
Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions
Despite the widespread adoption of Vision-Language Understanding (VLU) benchmarks such as VQA v2, OKVQA, A-OKVQA, GQA, VCR, SWAG, and VisualCOMET, our analysis reveals a pervasive issue affecting their integrity: these benchmarks contain samples where answers rely on assumptions unsupported by the provided context.
Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond
Vision-language (VL) understanding tasks evaluate models' comprehension of complex visual scenes through multiple-choice questions.
UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
However, we find visual and textual fine-grained information, e. g., keywords in the sentence and objects in the image, can be fairly informative for semantics understanding.
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer.
CoBIT: A Contrastive Bi-directional Image-Text Generation Model
The field of vision and language has witnessed a proliferation of pre-trained foundation models.
Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding
We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions.
Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense
Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described.
Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks
Experiments demonstrate that MAD leads to consistent gains in the low-shot, domain-shifted, and fully-supervised conditions on VCR, SNLI-VE, and VQA, achieving SOTA performance on VCR compared to other single models pretrained with image-text data.
Ranked #4 on
Visual Question Answering (VQA)
on VCR (Q-A) test
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51. 9%) and domain-shifted (up to 71. 3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only.
SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning
As for pre-training, a scene-graph-aware pre-training method is proposed to leverage structure knowledge extracted in the visual scene graph.
Graph-MLP: Node Classification without Message Passing in Graph
Graph Neural Network (GNN) has been demonstrated its effectiveness in dealing with non-Euclidean structural data.
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks.
Learning Visual Commonsense for Robust Scene Graph Generation
Scene graph generation models understand the scene through object and predicate recognition, but are prone to mistakes due to the challenges of perception in the wild.
Learning to Detect Head Movement in Unconstrained Remote Gaze Estimation in the Wild
To better demonstrate the advantage of our methods, we further propose a new benchmark dataset with the most rich distribution of head-gaze combination reflecting real-world scenarios.
Dual-Agent GANs for Photorealistic and Identity Preserving Profile Face Synthesis
In particular, we employ an off-the-shelf 3D face model as a simulator to generate profile face images with varying poses.
Ranked #1 on
Face Verification
on IJB-A
