Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 Jun 2023  ยท  Hang Zhang, Xin Li, Lidong Bing ยท

We present Video-LLaMA a multi-modal framework that empowers Large Language Models (LLMs) with the capability of understanding both visual and auditory content in the video. Video-LLaMA bootstraps cross-modal training from the frozen pre-trained visual and audio encoders and the frozen LLMs. Unlike previous works that complement LLMs to process the visual or audio signals only, Video-LLaMA enables video comprehension by tackling two challenges: (1) capturing the temporal changes in visual scenes, (2) integrating audio-visual signals. To counter the first challenge, we propose a Video Q-former to assemble a pre-trained image encoder into our video encoder and introduce a video-to-text generation task to learn video-language correspondence. For the second challenge, we leverage ImageBind, a universal embedding model aligning multiple modalities, as the pre-trained audio encoder and introduce an Audio Q-former on top of ImageBind to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual and audio encoders with LLM's embedding space, we first train Video-LLaMA on massive video/image-caption pairs and then tune our model with visual-instruction datasets of moderate amount but higher quality. We found Video-LLaMA shows the ability to perceive and comprehend video content and generate meaningful responses grounded in the visual and auditory information presented in the videos.

PDF Abstract

Results from the Paper


 Ranked #1 on Video-Text Retrieval on Test-of-Time (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Zero-Shot Video Question Answer ActivityNet-QA Video LLaMA Confidence Score 1.1 # 24
Accuracy 12.4 # 26
Zero-Shot Video Question Answer MSRVTT-QA Video LLaMA-7B Accuracy 29.6 # 28
Confidence Score 1.8 # 26
Zero-Shot Video Question Answer MSVD-QA Video LLaMA-7B Accuracy 51.6 # 25
Confidence Score 2.5 # 23
Video Question Answering MVBench VideoLLaMA Avg. 34.1 # 17
Video-Text Retrieval Test-of-Time Video-LLAMA 2-Class Accuracy 88.33 # 1
Video-based Generative Performance Benchmarking (Consistency) VideoInstruct Video LLaMA gpt-score 1.79 # 18
Video-based Generative Performance Benchmarking (Correctness of Information) VideoInstruct Video LLaMA gpt-score 1.96 # 18
Video-based Generative Performance Benchmarking VideoInstruct Video LLaMA Correctness of Information 1.96 # 21
Detail Orientation 2.18 # 21
Contextual Understanding 2.16 # 21
Temporal Understanding 1.82 # 21
Consistency 1.79 # 21
mean 1.98 # 23
Video-based Generative Performance Benchmarking (Temporal Understanding) VideoInstruct Video LLaMA gpt-score 1.82 # 18
Video-based Generative Performance Benchmarking (Detail Orientation)) VideoInstruct Video LLaMA gpt-score 2.18 # 18
Video-based Generative Performance Benchmarking (Contextual Understanding) VideoInstruct Video LLaMA gpt-score 2.16 # 18

Methods