Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at https://antoyang.github.io/vid2seq.html.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper


 Ranked #1 on Dense Video Captioning on ActivityNet Captions (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Dense Video Captioning ActivityNet Captions Vid2Seq METEOR 17 # 1
CIDEr 28 # 3
Video Captioning MSR-VTT Vid2Seq CIDEr 64.6 # 9
METEOR 30.8 # 8
Video Captioning MSVD Vid2Seq CIDEr 146.2 # 6
METEOR 45.3 # 4
Dense Video Captioning ViTT Vid2Seq SODA 0.135 # 2
CIDEr 43.5 # 2
METEOR 8.5 # 2
Dense Video Captioning YouCook2 Vid2Seq METEOR 9.3 # 2
CIDEr 47.1 # 2
SODA 7.9 # 2

Methods