UniVTG: Towards Unified Video-Language Temporal Grounding

Video Temporal Grounding (VTG), which aims to ground target clips from videos (such as consecutive intervals or disjoint shots) according to custom language queries (e.g., sentences or words), is key for video browsing on social media. Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels. In this paper, we propose to Unify the diverse VTG labels and tasks, dubbed UniVTG, along three directions: Firstly, we revisit a wide range of VTG labels and tasks and define a unified formulation. Based on this, we develop data annotation schemes to create scalable pseudo supervision. Secondly, we develop an effective and flexible grounding model capable of addressing each task and making full use of each label. Lastly, thanks to the unified framework, we are able to unlock temporal grounding pretraining from large-scale diverse labels and develop stronger grounding abilities e.g., zero-shot grounding. Extensive experiments on three tasks (moment retrieval, highlight detection and video summarization) across seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights, TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed framework. The codes are available at https://github.com/showlab/UniVTG.

PDF Abstract ICCV 2023 PDF ICCV 2023 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Moment Retrieval QVHighlights UniVTG mAP 35.47 # 24
R@1 IoU=0.5 58.86 # 25
R@1 IoU=0.7 40.86 # 24
mAP@0.5 57.60 # 21
mAP@0.75 35.59 # 20
Highlight Detection QVHighlights UniVTG (w/ PT) mAP 40.54 # 6
Hit@1 66.28 # 5
Highlight Detection QVHighlights UniVTG mAP 38.20 # 13
Hit@1 60.96 # 13
Moment Retrieval QVHighlights UniVTG (w/ PT) mAP 43.63 # 13
R@1 IoU=0.5 65.43 # 10
R@1 IoU=0.7 50.06 # 7
mAP@0.5 64.06 # 14
mAP@0.75 45.02 # 10
Natural Language Moment Retrieval TACoS UniVTG R@1,IoU=0.3 51.44 # 6
R@1,IoU=0.5 34.97 # 8
R@1,IoU=0.7 21.07 # 7
mIoU 35.76 # 6

Methods


No methods listed for this paper. Add relevant methods here