HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at:

PDF Abstract ICCV 2019 PDF ICCV 2019 Abstract


Introduced in the Paper:


Used in the Paper:

MSR-VTT YouCook2 DiDeMo LSMDC CrossTask
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Temporal Action Localization CrossTask Text-Video Embedding Recall 33.6 # 4
Video Retrieval LSMDC Text-Video Embedding text-to-video R@1 7.2 # 20
text-to-video R@5 19.6 # 18
text-to-video R@10 27.9 # 17
text-to-video Median Rank 40 # 14
Video Retrieval MSR-VTT Text-Video Embedding text-to-video R@1 14.9 # 16
text-to-video R@10 52.8 # 14
text-to-video Median Rank 9 # 10
video-to-text R@5 40.2 # 7
Video Retrieval MSR-VTT-1kA HT-Pretrained text-to-video R@1 14.9 # 31
text-to-video R@5 40.2 # 31
text-to-video R@10 52.8 # 31
text-to-video Median Rank 9 # 25
Video Retrieval MSR-VTT-1kA HT text-to-video R@1 12.1 # 32
text-to-video R@5 35.0 # 32
text-to-video R@10 48.0 # 32
text-to-video Median Rank 12 # 26
Video Retrieval YouCook2 Text-Video Embedding text-to-video Median Rank 24 # 6
text-to-video R@1 8.2 # 8
text-to-video R@10 35.3 # 8
text-to-video R@5 24.5 # 7


No methods listed for this paper. Add relevant methods here