MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization
In this work we present a new State-of-The-Art on the text-to-video retrieval task on MSR-VTT, LSMDC, MSVD, YouCook2 and TGIF obtained by a single model. Three different data sources are combined: weakly-supervised videos, crowd-labeled text-image pairs and text-video pairs. A careful analysis of available pre-trained networks helps to choose the best prior-knowledge ones. We introduce three-stage training procedure that provides high transfer knowledge efficiency and allows to use noisy datasets during training without prior knowledge degradation. Additionally, double positional encoding is used for better fusion of different modalities and a simple method for non-square inputs processing is suggested.
PDF Abstract
MS COCO
Flickr30k
ActivityNet
MSR-VTT
Conceptual Captions
MSVD
HowTo100M
Something-Something V2
YouCook2
TVQA
LSMDC
VATEX
TGIF
Results from the Paper
Ranked #1 on
Video Retrieval
on TGIF
(using extra training data)
