Search Results for author:
Found 4 papers, 2 papers with code
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA).
Ranked #1 on
Audio-Visual Captioning
on VALOR-32K
(using extra training data)
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
We show that only language-paired two-modality data is sufficient to connect all modalities.
MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning
Our method performs joint masking on image-text input and integrates both implicit and explicit targets for the masked signals to recover.
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources.
Ranked #1 on
Image Retrieval
on Localized Narratives
