TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Zero-Shot Transfer Image Classification	CN-ImageNet	AltCLIP	Accuracy (Private)	59.6	# 2
Zero-Shot Transfer Image Classification	CN-ImageNet-A	AltCLIP	Accuracy (Private)	58.5	# 1
Zero-Shot Transfer Image Classification	CN-ImageNet-R	AltCLIP	Accuracy (Private)	79.9	# 1
Zero-Shot Transfer Image Classification	CN-ImageNet-Sketch	AltCLIP	Accuracy (Private)	46.5	# 1
Zero-Shot Transfer Image Classification	CN-ImageNet V2	AltCLIP	Accuracy (Private)	50.9	# 1
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Image-to-text R@1	86	# 15
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Image-to-text R@5	98	# 16
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Image-to-text R@10	99.1	# 14
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Text-to-image R@1	72.5	# 15
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Text-to-image R@5	91.6	# 14
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Text-to-image R@10	95.4	# 12
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP	R@1	69.8	# 6
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP	R@5	89.9	# 7
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP	R@10	94.7	# 7
Zero-shot Text Retrieval	Flickr30k-CN	AltCLIP	R@1	84.8	# 2
Zero-shot Text Retrieval	Flickr30k-CN	AltCLIP	R@5	97.4	# 3
Zero-shot Text Retrieval	Flickr30k-CN	AltCLIP	R@10	98.8	# 3
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP(ViT-H/14)	R@1	74.5	# 4
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP(ViT-H/14)	R@5	92.0	# 4
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP(ViT-H/14)	R@10	95.5	# 4
Zero-shot Text Retrieval	Flickr30k-CN	Alt-CLIP(ViT-H/14)	R@1	88.9	# 1
Zero-shot Text Retrieval	Flickr30k-CN	Alt-CLIP(ViT-H/14)	R@5	98.5	# 1
Zero-shot Text Retrieval	Flickr30k-CN	Alt-CLIP(ViT-H/14)	R@10	99.5	# 1
Zero-Shot Transfer Image Classification	ImageNet	AltCLIP	Accuracy (Private)	74.5	# 19
Zero-Shot Transfer Image Classification	ImageNet-A	AltCLIP	Accuracy (Private)	69.5	# 12
Zero-Shot Transfer Image Classification	ImageNet-R	AltCLIP	Accuracy	87.2	# 11
Zero-Shot Transfer Image Classification	ImageNet-Sketch	AltCLIP	Accuracy (Private)	58.7	# 7
Zero-Shot Transfer Image Classification	ImageNet V2	AltCLIP	Accuracy (Private)	68.1	# 12
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	EN-Recall@10	95.4	# 3
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	ES-Recall@10	94.1	# 3
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	FR-Recall@10	92.9	# 3
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	ZH-Recall@10	95.1	# 3
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	KO-Recall@10	94.4	# 2
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	RU-Recall@10	91.8	# 3
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	JA-Recall@10	91.7	# 3
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	IT-Recall@10	94.2	# 3
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	EN-Recall@10	91.8	# 4
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	ES-Recall@10	89.1	# 4
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	FR-Recall@10	89.4	# 4
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	ZH-Recall@10	89.3	# 4
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	KO-Recall@10	82.1	# 4
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	RU-Recall@10	86.1	# 4
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	JA-Recall@10	81	# 4

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-cn-2)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-cn-2?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-cn-3)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-cn-3?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-cn-1)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-cn-1?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-cn-4)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-cn-4?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-text-retrieval-on-flickr30k-cn)](https://paperswithcode.com/sota/zero-shot-text-retrieval-on-flickr30k-cn?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-cn)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-cn?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-image-retrieval-on-xtd10)](https://paperswithcode.com/sota/zero-shot-image-retrieval-on-xtd10?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-image-retrieval-on-flickr30k-cn)](https://paperswithcode.com/sota/zero-shot-image-retrieval-on-flickr30k-cn?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-8)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-8?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-4)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-4?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-5)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-5?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-3)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-3?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-cross-modal-retrieval-on-flickr30k)](https://paperswithcode.com/sota/zero-shot-cross-modal-retrieval-on-flickr30k?p=altclip-altering-the-language-encoder-in-clip)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/altclip-altering-the-language-encoder-in-clip/zero-shot-transfer-image-classification-on-1)](https://paperswithcode.com/sota/zero-shot-transfer-image-classification-on-1?p=altclip-altering-the-language-encoder-in-clip)`

AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

12 Nov 2022 · Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu ·

In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model. Starting from the pre-trained multimodal representation model CLIP released by OpenAI, we altered its text encoder with a pre-trained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k-CN, COCO-CN and XTD. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

PDF Abstract

Code

Add Remove Mark official

flagai-open/flagai official

↳ Quickstart in

Spaces

3,814

Tasks

Add Remove

Contrastive Learning

Cross-Modal Retrieval

Image Classification

Image Retrieval

Image-to-Text Retrieval

Text-to-Image Generation

XLM-R

Zero-Shot Cross-Modal Retrieval

Zero-Shot Image Classification

Zero-shot Image Retrieval

Zero-shot Text Retrieval

Zero-Shot Transfer Image Classification

Zero-Shot Transfer Image Classification (CN)

Datasets

ImageNet

MS COCO

Kinetics

Oxford 102 Flower

Flickr30k

Kinetics 400

ImageNet-R

ImageNet-A

ImageNet-Sketch

Kinetics-600

Flickr30k-CNA XTD10 ImageNet_CN

Results from the Paper

Edit

Ranked #1 on Zero-Shot Transfer Image Classification on CN-ImageNet V2

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Zero-Shot Transfer Image Classification	CN-ImageNet	AltCLIP	Accuracy (Private)	59.6	# 2	Compare
Zero-Shot Transfer Image Classification	CN-ImageNet-A	AltCLIP	Accuracy (Private)	58.5	# 1	Compare
Zero-Shot Transfer Image Classification	CN-ImageNet-R	AltCLIP	Accuracy (Private)	79.9	# 1	Compare
Zero-Shot Transfer Image Classification	CN-ImageNet-Sketch	AltCLIP	Accuracy (Private)	46.5	# 1	Compare
Zero-Shot Transfer Image Classification	CN-ImageNet V2	AltCLIP	Accuracy (Private)	50.9	# 1	Compare
Zero-Shot Cross-Modal Retrieval	Flickr30k	AltCLIP	Image-to-text R@1	86	# 15	Compare
			Image-to-text R@5	98	# 16	Compare
			Image-to-text R@10	99.1	# 14	Compare
			Text-to-image R@1	72.5	# 15	Compare
			Text-to-image R@5	91.6	# 14	Compare
			Text-to-image R@10	95.4	# 12	Compare
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP	R@1	69.8	# 6	Compare
			R@5	89.9	# 7	Compare
			R@10	94.7	# 7	Compare
Zero-shot Text Retrieval	Flickr30k-CN	AltCLIP	R@1	84.8	# 2	Compare
			R@5	97.4	# 3	Compare
			R@10	98.8	# 3	Compare
Zero-shot Image Retrieval	Flickr30k-CN	AltCLIP(ViT-H/14)	R@1	74.5	# 4	Compare
			R@5	92.0	# 4	Compare
			R@10	95.5	# 4	Compare
Zero-shot Text Retrieval	Flickr30k-CN	Alt-CLIP(ViT-H/14)	R@1	88.9	# 1	Compare
			R@5	98.5	# 1	Compare
			R@10	99.5	# 1	Compare
Zero-Shot Transfer Image Classification	ImageNet	AltCLIP	Accuracy (Private)	74.5	# 19	Compare
Zero-Shot Transfer Image Classification	ImageNet-A	AltCLIP	Accuracy (Private)	69.5	# 12	Compare
Zero-Shot Transfer Image Classification	ImageNet-R	AltCLIP	Accuracy	87.2	# 11	Compare
Zero-Shot Transfer Image Classification	ImageNet-Sketch	AltCLIP	Accuracy (Private)	58.7	# 7	Compare
Zero-Shot Transfer Image Classification	ImageNet V2	AltCLIP	Accuracy (Private)	68.1	# 12	Compare
Zero-shot Image Retrieval	XTD10	AltCLIP(M9)	EN-Recall@10	95.4	# 3	Compare
			ES-Recall@10	94.1	# 3	Compare
			FR-Recall@10	92.9	# 3	Compare
			ZH-Recall@10	95.1	# 3	Compare
			KO-Recall@10	94.4	# 2	Compare
			RU-Recall@10	91.8	# 3	Compare
			JA-Recall@10	91.7	# 3	Compare
			IT-Recall@10	94.2	# 3	Compare
Zero-shot Image Retrieval	XTD10	M-CLIP(ViT-B32)	EN-Recall@10	91.8	# 4	Compare
			ES-Recall@10	89.1	# 4	Compare
			FR-Recall@10	89.4	# 4	Compare
			ZH-Recall@10	89.3	# 4	Compare
			KO-Recall@10	82.1	# 4	Compare
			RU-Recall@10	86.1	# 4	Compare
			JA-Recall@10	81	# 4	Compare

Methods

Add Remove

AltCLIP • AltDiffusion • CLIP • Diffusion • XLM-R

Edit Social Preview

AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove