VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning

26 Jun 2022  ·  Kashu Yamazaki, Sang Truong, Khoa Vo, Michael Kidd, Chase Rainwater, Khoa Luu, Ngan Le ·

In this paper, we leverage the human perceiving process, that involves vision and language interaction, to generate a coherent paragraph description of untrimmed videos. We propose vision-language (VL) features consisting of two modalities, i.e., (i) vision modality to capture global visual content of the entire scene and (ii) language modality to extract scene elements description of both human and non-human objects (e.g. animals, vehicles, etc), visual and non-visual elements (e.g. relations, activities, etc). Furthermore, we propose to train our proposed VLCap under a contrastive learning VL loss. The experiments and ablation studies on ActivityNet Captions and YouCookII datasets show that our VLCap outperforms existing SOTA methods on both accuracy and diversity metrics.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Captioning ActivityNet Captions VLCap (ae-test split) - Appearance + Language ROUGE-L 35.99 # 2
METEOR 17.48 # 2
BLEU4 13.38 # 3
CIDEr 31.29 # 2

Methods