Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

16 Nov 2023  Β·  Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, Li Yuan Β·

The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding. Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models. However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4 image benchmark toolkits. Additionally, our Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos. We aim for this work to provide modest insights into the multi-modal inputs for the LLM. Code address: \href{https://github.com/PKU-YuanGroup/Video-LLaVA}

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Zero-Shot Video Question Answer ActivityNet-QA Video-LLaVA Confidence Score 3.3 # 9
Accuracy 45.3 # 18
Video Question Answering ActivityNet-QA Video-LLaVA Accuracy 45.3 # 16
Confidence score 3.3 # 2
Visual Question Answering MM-Vet Video-LLaVA GPT-4 score 32.0 # 123
Zero-Shot Video Question Answer MSRVTT-QA Video-LLaVA-7B Accuracy 59.2 # 13
Confidence Score 3.5 # 5
Zero-Shot Video Question Answer MSVD-QA Video-LLaVA-7B Accuracy 70.7 # 13
Confidence Score 3.9 # 5
Zero-Shot Video Question Answer TGIF-QA Video-LLaVA-7B Accuracy 70.0 # 7
Confidence Score 4.0 # 6
Temporal Relation Extraction Vinoground Video-LLaVA-7B Text Score 24.8 # 15
Video Score 25.8 # 12
Group Score 6.6 # 16

Methods


No methods listed for this paper. Add relevant methods here