Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA

Long-form videos that span across wide temporal intervals are highly information redundant and contain multiple distinct events or entities that are often loosely related. Therefore, when performing long-form video question answering (LVQA), all information necessary to generate a correct response can often be contained within a small subset of frames. Recent literature explore the use of large language models (LLMs) in LVQA benchmarks, achieving exceptional performance, while relying on vision language models (VLMs) to convert all visual content within videos into natural language. Such VLMs often independently caption a large number of frames uniformly sampled from long videos, which is not efficient and can mostly be redundant. Questioning these decision choices, we explore optimal strategies for key-frame selection that can significantly reduce these redundancies, namely Hierarchical Keyframe Selector. Our proposed framework, LVNet, achieves state-of-the-art performance at a comparable caption scale across three benchmark LVQA datasets: EgoSchema, IntentQA, NExT-QA. The code can be found at https://github.com/jongwoopark7978/LVNet

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Video Question Answer EgoSchema (fullset) LVNet Accuracy 61.1 # 4
Zero-Shot Video Question Answer EgoSchema (subset) LVNet Accuracy 66.0 # 4
Zero-Shot Video Question Answer IntentQA LVNet Accuracy 71.1 # 1
Zero-Shot Video Question Answer NExT-QA LVNet(GPT-4o) Accuracy 72.9 # 3

Methods


No methods listed for this paper. Add relevant methods here