Search Results for author:
Found 13 papers, 10 papers with code
ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval
In this paper, we investigate the task of general conversational image retrieval on open-domain images.
Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining datasets.
Exploring the Design Space of Visual Context Representation in Video MLLMs
Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments.
OneDiff: A Generalist Model for Image Difference Captioning
This paper introduces the OneDiff model, a novel generalist approach that utilizes a robust vision-language model architecture, integrating a siamese image encoder with a Visual Delta Module.
Towards Event-oriented Long Video Understanding
Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench.
Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.
VL-Mamba: Exploring State Space Models for Multimodal Learning
The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.
Ranked #177 on
Visual Question Answering
on MM-Vet
SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models
In this paper, we present and delve into the self-consistency capability of LVLMs, a crucial aspect that reflects the models' ability to both generate informative captions for specific objects and subsequently utilize these captions to accurately re-identify the objects in a closed-loop process.
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions
To promote classic VG towards human intention interpretation, we propose a new intention-driven visual grounding (IVG) task and build a large-scale IVG dataset termed IntentionVG with free-form intention expressions.
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA).
Ranked #1 on
Image Captioning
on COCO Captions
(SPICE metric, using extra
training data)
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
We show that only language-paired two-modality data is sufficient to connect all modalities.
MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning
Our method performs joint masking on image-text input and integrates both implicit and explicit targets for the masked signals to recover.
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources.
Ranked #1 on
Image Retrieval
on Localized Narratives
