HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.

PDF Abstract ICCV 2023 PDF ICCV 2023 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Video Retrieval ActivityNet HiTeA text-to-video R@1 49.7 # 10
text-to-video R@5 77.1 # 9
text-to-video R@10 86.7 # 9
Video Retrieval DiDeMo HiTeA text-to-video R@1 56.5 # 6
text-to-video R@5 81.7 # 5
text-to-video R@10 89.7 # 4
Zero-Shot Video Retrieval DiDeMo HiTeA-17M text-to-video R@1 43.2 # 4
text-to-video R@5 69.3 # 4
text-to-video R@10 79.0 # 3
Zero-Shot Video Retrieval DiDeMo HiTeA-5M text-to-video R@1 36.1 # 7
text-to-video R@5 60.1 # 7
text-to-video R@10 70.3 # 5
Zero-Shot Video Retrieval LSMDC HiTeA-5M text-to-video R@1 15.5 # 5
text-to-video R@5 31.1 # 5
text-to-video R@10 39.8 # 5
Video Retrieval LSMDC HiTeA text-to-video R@1 28.7 # 9
text-to-video R@5 50.3 # 6
text-to-video R@10 59.0 # 6
Zero-Shot Video Retrieval LSMDC HiTeA-17M text-to-video R@1 18.3 # 3
text-to-video R@5 36.7 # 3
text-to-video R@10 44.2 # 3
Zero-Shot Video Retrieval MSR-VTT HiTeA-5M text-to-video R@1 29.9 # 13
text-to-video R@5 54.2 # 12
text-to-video R@10 62.9 # 12
Video Captioning MSR-VTT HiTeA CIDEr 65.1 # 8
METEOR 30.7 # 8
ROUGE-L 65.0 # 6
BLEU-4 49.2 # 7
Zero-Shot Video Retrieval MSR-VTT HiTeA-17M text-to-video R@1 34.4 # 9
text-to-video R@5 60.0 # 7
text-to-video R@10 69.9 # 7
Video Retrieval MSR-VTT-1kA HiTeA text-to-video R@1 46.8 # 24
text-to-video R@5 71.2 # 27
text-to-video R@10 81.9 # 26
Video Question Answering MSRVTT-MC HiTeA Accuracy 97.4 # 2
Visual Question Answering (VQA) MSRVTT-QA HiTeA Accuracy 0.459 # 12
Zero-Shot Learning MSRVTT-QA HiTeA Accuracy 21.7 # 1
Video Captioning MSVD HiTeA CIDEr 146.9 # 5
BLEU-4 71.0 # 3
METEOR 45.3 # 4
ROUGE-L 81.4 # 4
Zero-Shot Learning MSVD-QA HiTeA Accuracy 37.4 # 1
Visual Question Answering (VQA) MSVD-QA HiTeA Accuracy 0.556 # 9
Video Question Answering NExT-QA HiTeA Accuracy 63.1 # 1
Video Retrieval SSv2-label retrieval HiTeA text-to-video R@1 55.2 # 2
text-to-video R@5 89.1 # 2
text-to-video R@10 81.4 # 3
Video Retrieval SSv2-template retrieval HiTeA text-to-video R@1 85.6 # 2
text-to-video R@5 100 # 1
text-to-video R@10 100 # 1
TGIF-Transition TGIF-QA HiTeA Accuracy 98.8 # 3
Visual Question Answering (VQA) TGIF-QA HiTeA Accuracy 0.732 # 1
TGIF-Action TGIF-QA HiTeA Accuracy 97.2 # 2
TGIF-Frame TGIF-QA HiTeA Accuracy 73.2 # 6

Methods