AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

12 Nov 2022  ·  Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu ·

In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model. Starting from the pre-trained multimodal representation model CLIP released by OpenAI, we altered its text encoder with a pre-trained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k-CN, COCO-CN and XTD. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Transfer Image Classification CN-ImageNet AltCLIP Accuracy (Private) 59.6 # 1
Zero-Shot Transfer Image Classification CN-ImageNet-A AltCLIP Accuracy (Private) 58.5 # 1
Zero-Shot Transfer Image Classification CN-ImageNet-R AltCLIP Accuracy (Private) 79.9 # 1
Zero-Shot Transfer Image Classification CN-ImageNet-Sketch AltCLIP Accuracy (Private) 46.5 # 1
Zero-Shot Transfer Image Classification CN-ImageNet V2 AltCLIP Accuracy (Private) 50.9 # 1
Zero-Shot Cross-Modal Retrieval Flickr30k AltCLIP Image-to-text R@1 86 # 9
Image-to-text R@5 98 # 9
Image-to-text R@10 99.1 # 8
Text-to-image R@1 72.5 # 8
Text-to-image R@5 91.6 # 8
Text-to-image R@10 95.4 # 7
Zero-shot Text Retrieval Flickr30k-CN AltCLIP R@1 84.8 # 2
R@5 97.4 # 3
R@10 98.8 # 3
Zero-shot Text Retrieval Flickr30k-CN Alt-CLIP(ViT-H/14) R@1 88.9 # 1
R@5 98.5 # 1
R@10 99.5 # 1
Zero-shot Image Retrieval Flickr30k-CN AltCLIP R@1 69.8 # 3
R@5 89.9 # 4
R@10 94.7 # 4
Zero-shot Image Retrieval Flickr30k-CN AltCLIP(ViT-H/14) R@1 74.5 # 1
R@5 92.0 # 1
R@10 95.5 # 1
Zero-Shot Transfer Image Classification ImageNet AltCLIP Accuracy (Private) 74.5 # 9
Zero-Shot Transfer Image Classification ImageNet-A AltCLIP Accuracy (Private) 69.5 # 7
Zero-Shot Transfer Image Classification ImageNet-R AltCLIP Accuracy (Private) 87.2 # 7
Zero-Shot Transfer Image Classification ImageNet-Sketch AltCLIP Accuracy (Private) 58.7 # 3
Zero-Shot Transfer Image Classification ImageNet V2 AltCLIP Accuracy (Private) 68.1 # 7
Zero-shot Image Retrieval XTD10 AltCLIP(M9) EN-Recall@10 95.4 # 1
ES-Recall@10 94.1 # 1
FR-Recall@10 92.9 # 1
ZH-Recall@10 95.1 # 1
KO-Recall@10 94.4 # 1
RU-Recall@10 91.8 # 1
JA-Recall@10 91.7 # 1
IT-Recall@10 94.2 # 1

Methods