Revealing Single Frame Bias for Video-and-Language Learning

7 Jun 2022  ·  Jie Lei, Tamara L. Berg, Mohit Bansal ·

Training an effective video-and-language model intuitively requires multiple frames as model inputs. However, it is unclear whether using multiple frames is beneficial to downstream tasks, and if yes, whether the performance gain is worth the drastically-increased computation and memory costs resulting from using more frames. In this work, we explore single-frame models for video-and-language learning. On a diverse set of video-and-language tasks (including text-to-video retrieval and video question answering), we show the surprising result that, with large-scale pre-training and a proper frame ensemble strategy at inference time, a single-frame trained model that does not consider temporal information can achieve better performance than existing methods that use multiple frames for training. This result reveals the existence of a strong "static appearance bias" in popular video-and-language datasets. Therefore, to allow for a more comprehensive evaluation of video-and-language models, we propose two new retrieval tasks based on existing fine-grained action recognition datasets that encourage temporal modeling. Our code is available at https://github.com/jayleicn/singularity

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Zero-Shot Video Retrieval ActivityNet Singularity text-to-video R@1 30.6 # 3
text-to-video R@10 66.9 # 2
text-to-video R@5 55.6 # 2
Video Retrieval ActivityNet Singularity text-to-video R@1 47.1 # 8
text-to-video R@5 75.5 # 7
text-to-video R@10 85.5 # 6
Video Question Answering ActivityNet-QA Singularity Accuracy 0.431 # 6
Video Question Answering ActivityNet-QA Singularity-temporal Accuracy 0.441 # 4
Video Retrieval DiDeMo Singularity text-to-video R@1 53.9 # 5
text-to-video R@5 79.4 # 4
text-to-video R@10 86.9 # 4
Video Retrieval MSR-VTT-1kA Singularity text-to-video R@1 41.5 # 23
text-to-video R@5 68.7 # 23
text-to-video R@10 77 # 25
Video Question Answering MSRVTT-MC Singularity Accuracy 92.1 # 4
Video Question Answering MSRVTT-MC Singularity-temporal Accuracy 93.7 # 3
Video Question Answering MSRVTT-QA Singularity Accuracy 43.5 # 6
Video Question Answering MSRVTT-QA Singularity-temporal Accuracy 43.9 # 5
Video Retrieval SSv2-label retrieval Singularity-temporal text-to-video R@1 47.4 # 2
text-to-video R@5 75.9 # 2
text-to-video R@10 84 # 1
Video Retrieval SSv2-template retrieval Singularity-temporal text-to-video R@1 77.6 # 2
text-to-video R@5 96 # 2
text-to-video R@10 98.9 # 2

Methods


No methods listed for this paper. Add relevant methods here