CoCa: Contrastive Captioners are Image-Text Foundation Models

4 May 2022  Β·  Jiahui Yu, ZiRui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu Β·

Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Zero-Shot Cross-Modal Retrieval COCO 2014 CoCa Image-to-text R@1 66.3 # 8
Image-to-text R@5 86.2 # 9
Image-to-text R@10 91.8 # 9
Text-to-image R@1 51.2 # 6
Text-to-image R@5 74.2 # 8
Text-to-image R@10 82.0 # 9
Image Captioning COCO Captions CoCa BLEU-4 40.9 # 16
METEOR 33.9 # 1
CIDER 143.6 # 12
SPICE 24.7 # 10
Zero-Shot Cross-Modal Retrieval Flickr30k CoCa Image-to-text R@1 92.5 # 4
Image-to-text R@5 99.5 # 4
Image-to-text R@10 99.9 # 2
Text-to-image R@1 80.4 # 7
Text-to-image R@5 95.7 # 5
Text-to-image R@10 97.7 # 7
Zero-Shot Transfer Image Classification ImageNet CoCa Accuracy (Private) 86.3 # 3
Image Classification ImageNet CoCa (frozen) Top 1 Accuracy 90.60% # 7
Number of params 2100M # 967
Image Classification ImageNet CoCa (finetuned) Top 1 Accuracy 91.0% # 2
Number of params 2100M # 967
Zero-Shot Transfer Image Classification ImageNet-A CoCa Accuracy (Private) 90.2 # 1
Zero-Shot Transfer Image Classification ImageNet-R CoCa Accuracy 96.5 # 2
Zero-Shot Transfer Image Classification ImageNet-Sketch CoCa Accuracy (Private) 77.6 # 1
Zero-Shot Transfer Image Classification ImageNet V2 CoCa Accuracy (Private) 80.7 # 3
Action Classification Kinetics-400 CoCa (frozen) Acc@1 88.0 # 23
Action Classification Kinetics-400 CoCa (finetuned) Acc@1 88.9 # 16
Action Classification Kinetics-600 CoCa (finetuned) Top-1 Accuracy 89.4 # 15
Action Classification Kinetics-600 CoCa (frozen) Top-1 Accuracy 88.5 # 19
Action Classification Kinetics-700 CoCa (finetuned) Top-1 Accuracy 82.7 # 8
Action Classification Kinetics-700 CoCa (frozen) Top-1 Accuracy 81.1 # 10
Action Classification Moments in Time CoCa (frozen) Top 1 Accuracy 47.4 # 3
Action Classification Moments in Time CoCa (finetuned) Top 1 Accuracy 49.0 # 2
Video Retrieval MSR-VTT CoCa (zero-shot) text-to-video R@1 30.0 # 24
text-to-video R@5 52.4 # 25
text-to-video R@10 61.6 # 27
video-to-text R@1 49.9 # 8
video-to-text R@5 73.4 # 6
video-to-text R@10 81.4 # 6
Visual Reasoning NLVR2 Dev CoCa Accuracy 86.1 # 5
Visual Reasoning NLVR2 Test CoCa Accuracy 87.0 # 4
Zero-Shot Transfer Image Classification ObjectNet CoCa Accuracy (Private) 82.7 # 3
Image Classification ObjectNet CoCa Top-1 Accuracy 82.7 # 1
Visual Entailment SNLI-VE test CoCa Accuracy 87.1 # 3
Visual Entailment SNLI-VE val CoCa Accuracy 87.0 # 3
Visual Question Answering VQA v2 test-dev CoCa Accuracy 82.3 # 1

Methods


CLIP β€’ SimVLM