Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning

CVPR 2022  ·  Chia-Wen Kuo, Zsolt Kira ·

Significant progress has been made on visual captioning, largely relying on pre-trained features and later fixed object detectors that serve as rich inputs to auto-regressive models. A key limitation of such methods, however, is that the output of the model is conditioned only on the object detector's outputs. The assumption that such outputs can represent all necessary information is unrealistic, especially when the detector is transferred across datasets. In this work, we reason about the graphical model induced by this assumption, and propose to add an auxiliary input to represent missing information such as object relationships. We specifically propose to mine attributes and relationships from the Visual Genome dataset and condition the captioning model on them. Crucially, we propose (and show to be important) the use of a multi-modal pre-trained model (CLIP) to retrieve such contextual descriptions. Further, object detector models are frozen and do not have sufficient richness to allow the captioning model to properly ground them. As a result, we propose to condition both the detector and description outputs on the image, and show qualitatively and quantitatively that this can improve grounding. We validate our method on image captioning, perform thorough analyses of each component and importance of the pre-trained multi-modal model, and demonstrate significant improvements over the current state of the art, specifically +7.5% in CIDEr and +1.3% in BLEU-4 metrics.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Captioning COCO Captions Xmodal-Ctx BLEU-4 41.4 # 12
METEOR 30.4 # 14
ROUGE-L 60.4 # 3
CIDER 139.9 # 19
SPICE 24.0 # 16
BLEU-1 83.4 # 3
Image Captioning COCO Captions Xmodal-Ctx + OSCAR BLEU-4 41.3 # 13
CIDER 142.2 # 14
SPICE 24.9 # 9
Image Captioning COCO Captions Xmodal-Ctx BLEU-4 39.7 # 21
METEOR 30.0 # 16
ROUGE-L 59.5 # 5
CIDER 135.9 # 21
SPICE 23.7 # 17
BLEU-1 81.5 # 4

Methods


No methods listed for this paper. Add relevant methods here