Search Results for author:
Found 12 papers, 5 papers with code
iMOVE: Instance-Motion-Aware Video Understanding
Enhancing the fine-grained instance spatiotemporal motion perception capabilities of Video Large Language Models is crucial for improving their temporal and general video understanding.
MIBench: Evaluating Multimodal Large Language Models over Multiple Images
However, most existing MLLMs and benchmarks primarily focus on single-image input scenarios, leaving the performance of MLLMs when handling realistic multiple images underexplored.
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
In vision-language pre-training (VLP), masked image modeling (MIM) has recently been introduced for fine-grained cross-modal alignment.
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval
In this work, we propose the UNIFY framework, which learns lexicon representations to capture fine-grained semantics and combines the strengths of latent and lexicon representations for video-text retrieval.
mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model
In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs.
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks
In addition, to facilitate a comprehensive evaluation of video-language models, we carefully build the largest human-annotated Chinese benchmarks covering three popular video-language tasks of cross-modal retrieval, video captioning, and video category classification.
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github. com/X-PLUG/mPLUG-Owl.
Ranked #1 on
Visual Question Answering (VQA)
on HallusionBench
(Question Pair Acc metric)
Visual Question Answering (VQA)
Zero-Shot Video Question Answer
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement.
Ranked #1 on
Video Captioning
on MSR-VTT
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
The datasets will be released to facilitate the development of video captioning metrics.
A Simple and Strong Baseline for Universal Targeted Attacks on Siamese Visual Tracking
In this paper, we show the existence of universal perturbations that can enable the targeted attack, e. g., forcing a tracker to follow the ground-truth trajectory with specified offsets, to be video-agnostic and free from inference in a network.
Object Relational Graph with Teacher-Recommended Learning for Video Captioning
In this paper, we propose a complete video captioning system including both a novel model and an effective training strategy.
Ranked #9 on
Video Captioning
on VATEX
(using extra training data)
VATEX Captioning Challenge 2019: Multi-modal Information Fusion and Multi-stage Training Strategy for Video Captioning
Multi-modal information is essential to describe what has happened in a video.
