TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Visual Question Answering (VQA)	InfiMM-Eval	CogVLM-Chat	Overall score	37.16	# 4
Visual Question Answering (VQA)	InfiMM-Eval	CogVLM-Chat	Deductive	36.75	# 4
Visual Question Answering (VQA)	InfiMM-Eval	CogVLM-Chat	Abductive	47.88	# 4
Visual Question Answering (VQA)	InfiMM-Eval	CogVLM-Chat	Analogical	28.75	# 3
Visual Question Answering (VQA)	InfiMM-Eval	CogVLM-Chat	Params	17B	# 1
Long-Context Understanding	MMNeedle	CogVLM-17B	1 Image, 2*2 Stitching, Exact Accuracy	0	# 11
Long-Context Understanding	MMNeedle	CogVLM-17B	1 Image, 4*4 Stitching, Exact Accuracy	0.1	# 11
Long-Context Understanding	MMNeedle	CogVLM-17B	1 Image, 8*8 Stitching, Exact Accuracy	0.3	# 10
Long-Context Understanding	MMNeedle	CogVLM-17B	10 Images, 1*1 Stitching, Exact Accuracy	0	# 7
Long-Context Understanding	MMNeedle	CogVLM-17B	10 Images, 2*2 Stitching, Exact Accuracy	0	# 7
Long-Context Understanding	MMNeedle	CogVLM-17B	10 Images, 4*4 Stitching, Exact Accuracy	0	# 6
Long-Context Understanding	MMNeedle	CogVLM-17B	10 Images, 8*8 Stitching, Exact Accuracy	0	# 3
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	1 Image, 2*2 Stitching, Exact Accuracy	7.3	# 8
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	1 Image, 4*4 Stitching, Exact Accuracy	0.9	# 9
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	1 Image, 8*8 Stitching, Exact Accuracy	0.1	# 11
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	10 Images, 1*1 Stitching, Exact Accuracy	0	# 7
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	10 Images, 2*2 Stitching, Exact Accuracy	0	# 7
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	10 Images, 4*4 Stitching, Exact Accuracy	0	# 6
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	10 Images, 8*8 Stitching, Exact Accuracy	0	# 3
Visual Question Answering	MM-Vet	CogVLM(Vicuna-7B)	GPT-4 score	52.8	# 51
Visual Question Answering	MM-Vet	CogVLM(Vicuna-7B)	Params	17B	# 1
Visual Question Answering	MM-Vet	GLM4 Vision	GPT-4 score	63.9	# 25
Visual Question Answering	MM-Vet v2	CogVLM-Chat	GPT-4 score	45.1±0.2	# 14
FS-MEVQA	SME	GLM-4V	BLEU-4	14.45	# 5
FS-MEVQA	SME	GLM-4V	METEOR	17.53	# 6
FS-MEVQA	SME	GLM-4V	ROUGE-L	24.28	# 6
FS-MEVQA	SME	GLM-4V	CIDEr	127.37	# 5
FS-MEVQA	SME	GLM-4V	SPICE	17.70	# 5
FS-MEVQA	SME	GLM-4V	Detection	0.89	# 5
FS-MEVQA	SME	GLM-4V	ACC	34.23	# 5
FS-MEVQA	SME	GLM-4V	#Learning Samples (N)	16	# 1

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cogvlm-visual-expert-for-pretrained-language/visual-question-answering-vqa-on-core-mm)](https://paperswithcode.com/sota/visual-question-answering-vqa-on-core-mm?p=cogvlm-visual-expert-for-pretrained-language)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cogvlm-visual-expert-for-pretrained-language/fs-mevqa-on-sme)](https://paperswithcode.com/sota/fs-mevqa-on-sme?p=cogvlm-visual-expert-for-pretrained-language)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cogvlm-visual-expert-for-pretrained-language/long-context-understanding-on-mmneedle)](https://paperswithcode.com/sota/long-context-understanding-on-mmneedle?p=cogvlm-visual-expert-for-pretrained-language)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cogvlm-visual-expert-for-pretrained-language/visual-question-answering-on-mm-vet-v2)](https://paperswithcode.com/sota/visual-question-answering-on-mm-vet-v2?p=cogvlm-visual-expert-for-pretrained-language)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cogvlm-visual-expert-for-pretrained-language/visual-question-answering-on-mm-vet)](https://paperswithcode.com/sota/visual-question-answering-on-mm-vet?p=cogvlm-visual-expert-for-pretrained-language)`

CogVLM: Visual Expert for Pretrained Language Models

6 Nov 2023 · Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang ·

We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular shallow alignment method which maps image features into the input space of language model, CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. As a result, CogVLM enables deep fusion of vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X 55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.

PDF Abstract

Code

Add Remove Mark official

thudm/cogvlm official

6,273

THUDM/CogAgent

↳ Quickstart in

Spaces

508

MS-P3/code5

2024-MindSpore-1/Code2

Tasks

Add Remove

1 Image, 2*2 Stitching

FS-MEVQA

Image Retrieval

Language Modeling

Language Modelling

Long-Context Understanding

TextVQA

Visual Question Answering

Visual Question Answering (VQA)

Datasets

MS COCO

Visual Question Answering

MMLU

RefCOCO

TextVQA

OK-VQA

MMBench

ScienceQA

MM-Vet

NoCaps

MathVista

SEED-Bench

Visual7W LLaVA-Bench TextCaps

InfiMM-Eval

MM-Vet v2

MMNeedle

SME

Results from the Paper

Edit

Ranked #4 on Visual Question Answering (VQA) on InfiMM-Eval

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Visual Question Answering (VQA)	InfiMM-Eval	CogVLM-Chat	Overall score	37.16	# 4	Compare
			Deductive	36.75	# 4	Compare
			Abductive	47.88	# 4	Compare
			Analogical	28.75	# 3	Compare
			Params	17B	# 1	Compare
Long-Context Understanding	MMNeedle	CogVLM-17B	1 Image, 2*2 Stitching, Exact Accuracy	0	# 11	Compare
			1 Image, 4*4 Stitching, Exact Accuracy	0.1	# 11	Compare
			1 Image, 8*8 Stitching, Exact Accuracy	0.3	# 10	Compare
			10 Images, 1*1 Stitching, Exact Accuracy	0	# 7	Compare
			10 Images, 2*2 Stitching, Exact Accuracy	0	# 7	Compare
			10 Images, 4*4 Stitching, Exact Accuracy	0	# 6	Compare
			10 Images, 8*8 Stitching, Exact Accuracy	0	# 3	Compare
Long-Context Understanding	MMNeedle	CogVLM2-Llama-3	1 Image, 2*2 Stitching, Exact Accuracy	7.3	# 8	Compare
			1 Image, 4*4 Stitching, Exact Accuracy	0.9	# 9	Compare
			1 Image, 8*8 Stitching, Exact Accuracy	0.1	# 11	Compare
			10 Images, 1*1 Stitching, Exact Accuracy	0	# 7	Compare
			10 Images, 2*2 Stitching, Exact Accuracy	0	# 7	Compare
			10 Images, 4*4 Stitching, Exact Accuracy	0	# 6	Compare
			10 Images, 8*8 Stitching, Exact Accuracy	0	# 3	Compare
Visual Question Answering	MM-Vet	CogVLM(Vicuna-7B)	GPT-4 score	52.8	# 51	Compare
Visual Question Answering	MM-Vet	CogVLM(Vicuna-7B)	Params	17B	# 1	Compare
Visual Question Answering	MM-Vet	GLM4 Vision	GPT-4 score	63.9	# 25	Compare
Visual Question Answering	MM-Vet v2	CogVLM-Chat	GPT-4 score	45.1±0.2	# 14	Compare
FS-MEVQA	SME	GLM-4V	BLEU-4	14.45	# 5	Compare
			METEOR	17.53	# 6	Compare
			ROUGE-L	24.28	# 6	Compare
			CIDEr	127.37	# 5	Compare
			SPICE	17.70	# 5	Compare
			Detection	0.89	# 5	Compare
			ACC	34.23	# 5	Compare
			#Learning Samples (N)	16	# 1	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

CogVLM: Visual Expert for Pretrained Language Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove