QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries

20 Jul 2021  ·  Jie Lei, Tamara L. Berg, Mohit Bansal ·

Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, MomentDETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr

PDF Abstract


Introduced in the Paper:


Used in the Paper:

HowTo100M Charades-STA

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Moment Retrieval Charades-STA Moment-DETR w/ PT (on 10K HowTo100M videos) R@1 IoU=0.5 55.65 # 4
R@1 IoU=0.7 34.17 # 3
Moment Retrieval Charades-STA Moment-DETR R@1 IoU=0.5 53.63 # 5
R@1 IoU=0.7 31.37 # 5
Highlight Detection QVHighlights Moment-DETR w/ PT mAP 37.43 # 8
Hit@1 60.17 # 7
Moment Retrieval QVHighlights Moment-DETR w/ PT mAP 36.14 # 7
R@1 IoU=0.5 59.78 # 8
R@1 IoU=0.7 40.33 # 9
mAP@0.5 60.51 # 6
mAP@0.75 35.36 # 7


No methods listed for this paper. Add relevant methods here