MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding. The model is capable of processing both temporal visual and textual data, making it adept at understanding the complexities of videos. Building upon the success of MiniGPT-v2, which excelled in translating visual features into the LLM space for single images and achieved impressive results on various image-text benchmarks, this paper extends the model's capabilities to process a sequence of frames, enabling it to comprehend videos. MiniGPT4-video does not only consider visual content but also incorporates textual conversations, allowing the model to effectively answer queries involving both visual and text components. The proposed model outperforms existing state-of-the-art methods, registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF, and TVQA benchmarks respectively. Our models and code have been made publicly available here https://vision-cair.github.io/MiniGPT4-video/

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Video Question Answer ActivityNet-QA MiniGPT4-video-7B Accuracy 46.3 # 12
Zero-Shot Video Question Answer MSRVTT-QA MiniGPT4-video-7B Accuracy 59.73 # 9
Zero-Shot Video Question Answer MSVD-QA MiniGPT4-video-7B Accuracy 73.92 # 7
Zero-Shot Video Question Answer TGIF-QA MiniGPT4-video-7B Accuracy 72.22 # 4
Zero-Shot Video Question Answer TVQA MiniGPT4-video-7B Accuracy 54.21 # 3
Video-based Generative Performance Benchmarking (Correctness of Information) VideoInstruct MiniGPT4-video-7B gpt-score 3.08 # 5
Video-based Generative Performance Benchmarking (Detail Orientation)) VideoInstruct MiniGPT4-video-7B gpt-score 3.02 # 5
Video-based Generative Performance Benchmarking (Contextual Understanding) VideoInstruct MiniGPT4-video-7B gpt-score 3.57 # 5
Video-based Generative Performance Benchmarking (Temporal Understanding) VideoInstruct MiniGPT4-video-7B gpt-score 2.65 # 5
Video-based Generative Performance Benchmarking (Consistency) VideoInstruct MiniGPT4-video-7B gpt-score 2.67 # 6

Methods


No methods listed for this paper. Add relevant methods here