ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

22 Jan 2020  ·  Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, Arun Sacheti ·

In this paper, we introduce a new vision-language pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them. The model is pre-trained on four tasks simultaneously: Masked Language Modeling (MLM), Masked Object Classification (MOC), Masked Region Feature Regression (MRFR), and Image Text Matching (ITM). To further enhance the pre-training quality, we have collected a Large-scale weAk-supervised Image-Text (LAIT) dataset from Web. We first pre-train the model on this dataset, then conduct a second stage pre-training on Conceptual Captions and SBU Captions. Our experiments show that multi-stage pre-training strategy outperforms single-stage pre-training. We also fine-tune and evaluate our pre-trained ImageBERT model on image retrieval and text retrieval tasks, and achieve new state-of-the-art results on both MSCOCO and Flickr30k datasets.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Cross-Modal Retrieval COCO 2014 ImageBERT Image-to-text R@1 44.0 # 15
Image-to-text R@5 71.2 # 15
Image-to-text R@10 80.4 # 14
Text-to-image R@1 32.3 # 15
Text-to-image R@5 59.0 # 15
Text-to-image R@10 70.2 # 14
Zero-Shot Cross-Modal Retrieval Flickr30k ImageBERT Image-to-text R@1 70.7 # 18
Image-to-text R@5 90.2 # 19
Image-to-text R@10 94.0 # 17
Text-to-image R@1 54.3 # 19
Text-to-image R@5 79.6 # 19
Text-to-image R@10 87.5 # 17


No methods listed for this paper. Add relevant methods here