TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Visual Question Answering (VQA)	IconQA	UNITER	Sub-tasks (Img.)	78.71	# 4
Visual Question Answering (VQA)	IconQA	UNITER	Sub-tasks (Txt.)	72.39	# 3
Visual Question Answering (VQA)	IconQA	UNITER	Sub-tasks (Blank)	78.53	# 4
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Geo.)	81.31	# 5
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Cou.)	71.01	# 4
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Com.)	83.67	# 3
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Spa.)	48.34	# 8
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Sce.)	61.25	# 7
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Pat.)	60.81	# 3
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Tim.)	69.77	# 3
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Fra.)	78.37	# 2
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Est.)	99.41	# 2
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Alg.)	49.18	# 7
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Mea.)	99.38	# 2
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Sen.)	86.10	# 2
Visual Question Answering (VQA)	IconQA	UNITER	Reasoning (Pro.)	87.84	# 2
Visual Question Answering (VQA)	IconQA	DFAF	Sub-tasks (Img.)	77.72	# 5
Visual Question Answering (VQA)	IconQA	DFAF	Sub-tasks (Txt.)	72.17	# 5
Visual Question Answering (VQA)	IconQA	DFAF	Sub-tasks (Blank)	78.28	# 5
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Geo.)	81.80	# 4
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Cou.)	70.68	# 6
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Com.)	81.69	# 7
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Spa.)	51.42	# 5
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Sce.)	67.01	# 2
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Pat.)	56.60	# 6
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Tim.)	67.72	# 7
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Fra.)	77.60	# 3
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Est.)	99.02	# 5
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Alg.)	50.27	# 5
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Mea.)	98.83	# 6
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Sen.)	84.11	# 6
Visual Question Answering (VQA)	IconQA	DFAF	Reasoning (Pro.)	85.70	# 5
Visual Question Answering (VQA)	IconQA	ViT	Sub-tasks (Img.)	79.15	# 3
Visual Question Answering (VQA)	IconQA	ViT	Sub-tasks (Txt.)	72.34	# 4
Visual Question Answering (VQA)	IconQA	ViT	Sub-tasks (Blank)	78.92	# 3
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Geo.)	82.60	# 2
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Cou.)	70.84	# 5
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Com.)	82.12	# 5
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Spa.)	54.64	# 2
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Sce.)	68.80	# 1
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Pat.)	58.46	# 5
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Tim.)	68.66	# 4
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Fra.)	77.41	# 4
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Est.)	98.95	# 7
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Alg.)	51.10	# 2
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Mea.)	98.76	# 7
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Sen.)	84.72	# 4
Visual Question Answering (VQA)	IconQA	ViT	Reasoning (Pro.)	86.07	# 4
Visual Question Answering (VQA)	IconQA	ViLT	Sub-tasks (Img.)	79.67	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Sub-tasks (Txt.)	72.69	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Sub-tasks (Blank)	79.27	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Geo.)	82.61	# 1
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Cou.)	71.13	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Com.)	84.95	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Spa.)	53.38	# 3
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Sce.)	66.72	# 4
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Pat.)	59.22	# 4
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Tim.)	69.99	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Fra.)	75.81	# 6
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Est.)	99.02	# 5
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Alg.)	50.55	# 4
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Mea.)	98.91	# 5
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Sen.)	86.10	# 2
Visual Question Answering (VQA)	IconQA	ViLT	Reasoning (Pro.)	87.65	# 3
Visual Question Answering (VQA)	IconQA	Q-Only	Sub-tasks (Img.)	41.64	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Sub-tasks (Txt.)	36.86	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Sub-tasks (Blank)	28.45	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Geo.)	38.03	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Cou.)	33.63	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Com.)	48.19	# 10
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Spa.)	37.14	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Sce.)	35.37	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Pat.)	33.66	# 12
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Tim.)	48.09	# 10
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Fra.)	33.06	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Est.)	40.46	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Alg.)	28.02	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Mea.)	38.07	# 11
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Sen.)	45.25	# 10
Visual Question Answering (VQA)	IconQA	Q-Only	Reasoning (Pro.)	40.76	# 10
Visual Question Answering (VQA)	IconQA	BAN	Sub-tasks (Img.)	76.33	# 8
Visual Question Answering (VQA)	IconQA	BAN	Sub-tasks (Txt.)	70.82	# 7
Visual Question Answering (VQA)	IconQA	BAN	Sub-tasks (Blank)	75.54	# 7
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Geo.)	79.99	# 8
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Cou.)	67.56	# 8
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Com.)	82.12	# 5
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Spa.)	53.20	# 4
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Sce.)	66.92	# 3
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Pat.)	55.67	# 7
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Tim.)	66.50	# 9
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Fra.)	73.77	# 8
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Est.)	97.06	# 9
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Alg.)	47.46	# 8
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Mea.)	96.50	# 9
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Sen.)	82.12	# 8
Visual Question Answering (VQA)	IconQA	BAN	Reasoning (Pro.)	82.45	# 8
Visual Question Answering (VQA)	IconQA	I-Only	Sub-tasks (Img.)	41.56	# 12
Visual Question Answering (VQA)	IconQA	I-Only	Sub-tasks (Txt.)	36.02	# 12
Visual Question Answering (VQA)	IconQA	I-Only	Sub-tasks (Blank)	46.65	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Geo.)	38.71	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Cou.)	37.64	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Com.)	45.26	# 11
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Spa.)	37.52	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Sce.)	35.47	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Pat.)	36.29	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Tim.)	47.37	# 11
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Fra.)	32.48	# 12
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Est.)	62.29	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Alg.)	31.73	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Mea.)	64.02	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Sen.)	45.25	# 10
Visual Question Answering (VQA)	IconQA	I-Only	Reasoning (Pro.)	37.51	# 12
Visual Question Answering (VQA)	IconQA	ViLBERT	Sub-tasks (Img.)	76.66	# 7
Visual Question Answering (VQA)	IconQA	ViLBERT	Sub-tasks (Txt.)	70.47	# 8
Visual Question Answering (VQA)	IconQA	ViLBERT	Sub-tasks (Blank)	77.08	# 6
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Geo.)	80.05	# 7
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Cou.)	71.05	# 3
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Com.)	75.60	# 9
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Spa.)	49.46	# 7
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Sce.)	58.52	# 8
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Pat.)	62.78	# 2
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Tim.)	66.72	# 8
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Fra.)	74.09	# 7
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Est.)	99.22	# 3
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Alg.)	50.62	# 3
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Mea.)	99.07	# 3
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Sen.)	81.78	# 9
Visual Question Answering (VQA)	IconQA	ViLBERT	Reasoning (Pro.)	70.94	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Sub-tasks (Img.)	75.92	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Sub-tasks (Txt.)	68.51	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Sub-tasks (Blank)	73.03	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Geo.)	80.07	# 6
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Cou.)	65.01	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Com.)	80.65	# 8
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Spa.)	45.78	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Sce.)	58.22	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Pat.)	55.01	# 8
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Tim.)	68.28	# 5
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Fra.)	72.43	# 9
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Est.)	99.54	# 1
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Alg.)	50.00	# 6
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Mea.)	99.46	# 1
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Sen.)	84.54	# 5
Visual Question Answering (VQA)	IconQA	Top-Down	Reasoning (Pro.)	83.75	# 7
Visual Question Answering (VQA)	IconQA	MCAN	Sub-tasks (Img.)	77.36	# 6
Visual Question Answering (VQA)	IconQA	MCAN	Sub-tasks (Txt.)	71.25	# 6
Visual Question Answering (VQA)	IconQA	MCAN	Sub-tasks (Blank)	74.52	# 8
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Geo.)	79.86	# 9
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Cou.)	68.94	# 7
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Com.)	82.73	# 4
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Spa.)	49.70	# 6
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Sce.)	62.49	# 5
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Pat.)	54.79	# 9
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Tim.)	68.00	# 6
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Fra.)	76.20	# 5
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Est.)	99.08	# 4
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Alg.)	47.32	# 9
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Mea.)	98.99	# 4
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Sen.)	83.25	# 7
Visual Question Answering (VQA)	IconQA	MCAN	Reasoning (Pro.)	84.87	# 6
Visual Question Answering (VQA)	IconQA	Patch-TRM	Sub-tasks (Img.)	82.66	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Sub-tasks (Txt.)	75.19	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Sub-tasks (Blank)	83.62	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Geo.)	81.87	# 3
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Cou.)	77.81	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Com.)	87.00	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Spa.)	55.62	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Sce.)	62.39	# 6
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Pat.)	68.75	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Tim.)	77.98	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Fra.)	82.13	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Est.)	98.24	# 8
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Alg.)	56.73	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Mea.)	97.98	# 8
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Sen.)	92.49	# 1
Visual Question Answering (VQA)	IconQA	Patch-TRM	Reasoning (Pro.)	95.73	# 1
Visual Question Answering (VQA)	IconQA	Random	Sub-tasks (Img.)	41.70	# 10
Visual Question Answering (VQA)	IconQA	Random	Sub-tasks (Txt.)	36.87	# 10
Visual Question Answering (VQA)	IconQA	Random	Sub-tasks (Blank)	0.29	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Geo.)	30.30	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Cou.)	18.38	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Com.)	41.20	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Spa.)	36.49	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Sce.)	34.25	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Pat.)	34.81	# 11
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Tim.)	35.82	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Fra.)	34.84	# 10
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Est.)	3.62	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Alg.)	11.12	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Mea.)	0.36	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Sen.)	45.16	# 12
Visual Question Answering (VQA)	IconQA	Random	Reasoning (Pro.)	38.81	# 11

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/iconqa-a-new-benchmark-for-abstract-diagram/visual-question-answering-on-iconqa)](https://paperswithcode.com/sota/visual-question-answering-on-iconqa?p=iconqa-a-new-benchmark-for-abstract-diagram)`

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

25 Oct 2021 · Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei zhang, Zhou Yu, Xiaodan Liang, Song-Chun Zhu ·

Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning. To facilitate potential IconQA models to learn semantic representations for icon images, we further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes. We conduct extensive user studies and blind experiments and reproduce a wide range of advanced VQA methods to benchmark the IconQA task. Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset. IconQA and Icon645 are available at https://iconqa.github.io.

PDF Abstract

Code

Add Remove Mark official

lupantech/iconqa official

Tasks

Add Remove

Arithmetic Reasoning

Mathematical Question Answering

Mathematical Reasoning

Math Word Problem Solving

Object Recognition

Question Answering

Visual Question Answering

Visual Question Answering (VQA)

Datasets

Introduced in the Paper:

IconQA

Icon645

Used in the Paper:

Visual Question Answering

CLEVR test

Visual Question Answering v2.0

NLVR

AI2D

DVQA

Results from the Paper

Edit

Ranked #1 on Visual Question Answering (VQA) on IconQA

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Visual Question Answering (VQA)	IconQA	UNITER	Sub-tasks (Img.)	78.71	# 4	Compare
			Sub-tasks (Txt.)	72.39	# 3	Compare
			Sub-tasks (Blank)	78.53	# 4	Compare
			Reasoning (Geo.)	81.31	# 5	Compare
			Reasoning (Cou.)	71.01	# 4	Compare
			Reasoning (Com.)	83.67	# 3	Compare
			Reasoning (Spa.)	48.34	# 8	Compare
			Reasoning (Sce.)	61.25	# 7	Compare
			Reasoning (Pat.)	60.81	# 3	Compare
			Reasoning (Tim.)	69.77	# 3	Compare
			Reasoning (Fra.)	78.37	# 2	Compare
			Reasoning (Est.)	99.41	# 2	Compare
			Reasoning (Alg.)	49.18	# 7	Compare
			Reasoning (Mea.)	99.38	# 2	Compare
			Reasoning (Sen.)	86.10	# 2	Compare
			Reasoning (Pro.)	87.84	# 2	Compare
Visual Question Answering (VQA)	IconQA	DFAF	Sub-tasks (Img.)	77.72	# 5	Compare
			Sub-tasks (Txt.)	72.17	# 5	Compare
			Sub-tasks (Blank)	78.28	# 5	Compare
			Reasoning (Geo.)	81.80	# 4	Compare
			Reasoning (Cou.)	70.68	# 6	Compare
			Reasoning (Com.)	81.69	# 7	Compare
			Reasoning (Spa.)	51.42	# 5	Compare
			Reasoning (Sce.)	67.01	# 2	Compare
			Reasoning (Pat.)	56.60	# 6	Compare
			Reasoning (Tim.)	67.72	# 7	Compare
			Reasoning (Fra.)	77.60	# 3	Compare
			Reasoning (Est.)	99.02	# 5	Compare
			Reasoning (Alg.)	50.27	# 5	Compare
			Reasoning (Mea.)	98.83	# 6	Compare
			Reasoning (Sen.)	84.11	# 6	Compare
			Reasoning (Pro.)	85.70	# 5	Compare
Visual Question Answering (VQA)	IconQA	ViT	Sub-tasks (Img.)	79.15	# 3	Compare
			Sub-tasks (Txt.)	72.34	# 4	Compare
			Sub-tasks (Blank)	78.92	# 3	Compare
			Reasoning (Geo.)	82.60	# 2	Compare
			Reasoning (Cou.)	70.84	# 5	Compare
			Reasoning (Com.)	82.12	# 5	Compare
			Reasoning (Spa.)	54.64	# 2	Compare
			Reasoning (Sce.)	68.80	# 1	Compare
			Reasoning (Pat.)	58.46	# 5	Compare
			Reasoning (Tim.)	68.66	# 4	Compare
			Reasoning (Fra.)	77.41	# 4	Compare
			Reasoning (Est.)	98.95	# 7	Compare
			Reasoning (Alg.)	51.10	# 2	Compare
			Reasoning (Mea.)	98.76	# 7	Compare
			Reasoning (Sen.)	84.72	# 4	Compare
			Reasoning (Pro.)	86.07	# 4	Compare
Visual Question Answering (VQA)	IconQA	ViLT	Sub-tasks (Img.)	79.67	# 2	Compare
			Sub-tasks (Txt.)	72.69	# 2	Compare
			Sub-tasks (Blank)	79.27	# 2	Compare
			Reasoning (Geo.)	82.61	# 1	Compare
			Reasoning (Cou.)	71.13	# 2	Compare
			Reasoning (Com.)	84.95	# 2	Compare
			Reasoning (Spa.)	53.38	# 3	Compare
			Reasoning (Sce.)	66.72	# 4	Compare
			Reasoning (Pat.)	59.22	# 4	Compare
			Reasoning (Tim.)	69.99	# 2	Compare
			Reasoning (Fra.)	75.81	# 6	Compare
			Reasoning (Est.)	99.02	# 5	Compare
			Reasoning (Alg.)	50.55	# 4	Compare
			Reasoning (Mea.)	98.91	# 5	Compare
			Reasoning (Sen.)	86.10	# 2	Compare
			Reasoning (Pro.)	87.65	# 3	Compare
Visual Question Answering (VQA)	IconQA	Q-Only	Sub-tasks (Img.)	41.64	# 11	Compare
			Sub-tasks (Txt.)	36.86	# 11	Compare
			Sub-tasks (Blank)	28.45	# 11	Compare
			Reasoning (Geo.)	38.03	# 11	Compare
			Reasoning (Cou.)	33.63	# 11	Compare
			Reasoning (Com.)	48.19	# 10	Compare
			Reasoning (Spa.)	37.14	# 11	Compare
			Reasoning (Sce.)	35.37	# 11	Compare
			Reasoning (Pat.)	33.66	# 12	Compare
			Reasoning (Tim.)	48.09	# 10	Compare
			Reasoning (Fra.)	33.06	# 11	Compare
			Reasoning (Est.)	40.46	# 11	Compare
			Reasoning (Alg.)	28.02	# 11	Compare
			Reasoning (Mea.)	38.07	# 11	Compare
			Reasoning (Sen.)	45.25	# 10	Compare
			Reasoning (Pro.)	40.76	# 10	Compare
Visual Question Answering (VQA)	IconQA	BAN	Sub-tasks (Img.)	76.33	# 8	Compare
			Sub-tasks (Txt.)	70.82	# 7	Compare
			Sub-tasks (Blank)	75.54	# 7	Compare
			Reasoning (Geo.)	79.99	# 8	Compare
			Reasoning (Cou.)	67.56	# 8	Compare
			Reasoning (Com.)	82.12	# 5	Compare
			Reasoning (Spa.)	53.20	# 4	Compare
			Reasoning (Sce.)	66.92	# 3	Compare
			Reasoning (Pat.)	55.67	# 7	Compare
			Reasoning (Tim.)	66.50	# 9	Compare
			Reasoning (Fra.)	73.77	# 8	Compare
			Reasoning (Est.)	97.06	# 9	Compare
			Reasoning (Alg.)	47.46	# 8	Compare
			Reasoning (Mea.)	96.50	# 9	Compare
			Reasoning (Sen.)	82.12	# 8	Compare
			Reasoning (Pro.)	82.45	# 8	Compare
Visual Question Answering (VQA)	IconQA	I-Only	Sub-tasks (Img.)	41.56	# 12	Compare
			Sub-tasks (Txt.)	36.02	# 12	Compare
			Sub-tasks (Blank)	46.65	# 10	Compare
			Reasoning (Geo.)	38.71	# 10	Compare
			Reasoning (Cou.)	37.64	# 10	Compare
			Reasoning (Com.)	45.26	# 11	Compare
			Reasoning (Spa.)	37.52	# 10	Compare
			Reasoning (Sce.)	35.47	# 10	Compare
			Reasoning (Pat.)	36.29	# 10	Compare
			Reasoning (Tim.)	47.37	# 11	Compare
			Reasoning (Fra.)	32.48	# 12	Compare
			Reasoning (Est.)	62.29	# 10	Compare
			Reasoning (Alg.)	31.73	# 10	Compare
			Reasoning (Mea.)	64.02	# 10	Compare
			Reasoning (Sen.)	45.25	# 10	Compare
			Reasoning (Pro.)	37.51	# 12	Compare
Visual Question Answering (VQA)	IconQA	ViLBERT	Sub-tasks (Img.)	76.66	# 7	Compare
			Sub-tasks (Txt.)	70.47	# 8	Compare
			Sub-tasks (Blank)	77.08	# 6	Compare
			Reasoning (Geo.)	80.05	# 7	Compare
			Reasoning (Cou.)	71.05	# 3	Compare
			Reasoning (Com.)	75.60	# 9	Compare
			Reasoning (Spa.)	49.46	# 7	Compare
			Reasoning (Sce.)	58.52	# 8	Compare
			Reasoning (Pat.)	62.78	# 2	Compare
			Reasoning (Tim.)	66.72	# 8	Compare
			Reasoning (Fra.)	74.09	# 7	Compare
			Reasoning (Est.)	99.22	# 3	Compare
			Reasoning (Alg.)	50.62	# 3	Compare
			Reasoning (Mea.)	99.07	# 3	Compare
			Reasoning (Sen.)	81.78	# 9	Compare
			Reasoning (Pro.)	70.94	# 9	Compare
Visual Question Answering (VQA)	IconQA	Top-Down	Sub-tasks (Img.)	75.92	# 9	Compare
			Sub-tasks (Txt.)	68.51	# 9	Compare
			Sub-tasks (Blank)	73.03	# 9	Compare
			Reasoning (Geo.)	80.07	# 6	Compare
			Reasoning (Cou.)	65.01	# 9	Compare
			Reasoning (Com.)	80.65	# 8	Compare
			Reasoning (Spa.)	45.78	# 9	Compare
			Reasoning (Sce.)	58.22	# 9	Compare
			Reasoning (Pat.)	55.01	# 8	Compare
			Reasoning (Tim.)	68.28	# 5	Compare
			Reasoning (Fra.)	72.43	# 9	Compare
			Reasoning (Est.)	99.54	# 1	Compare
			Reasoning (Alg.)	50.00	# 6	Compare
			Reasoning (Mea.)	99.46	# 1	Compare
			Reasoning (Sen.)	84.54	# 5	Compare
			Reasoning (Pro.)	83.75	# 7	Compare
Visual Question Answering (VQA)	IconQA	MCAN	Sub-tasks (Img.)	77.36	# 6	Compare
			Sub-tasks (Txt.)	71.25	# 6	Compare
			Sub-tasks (Blank)	74.52	# 8	Compare
			Reasoning (Geo.)	79.86	# 9	Compare
			Reasoning (Cou.)	68.94	# 7	Compare
			Reasoning (Com.)	82.73	# 4	Compare
			Reasoning (Spa.)	49.70	# 6	Compare
			Reasoning (Sce.)	62.49	# 5	Compare
			Reasoning (Pat.)	54.79	# 9	Compare
			Reasoning (Tim.)	68.00	# 6	Compare
			Reasoning (Fra.)	76.20	# 5	Compare
			Reasoning (Est.)	99.08	# 4	Compare
			Reasoning (Alg.)	47.32	# 9	Compare
			Reasoning (Mea.)	98.99	# 4	Compare
			Reasoning (Sen.)	83.25	# 7	Compare
			Reasoning (Pro.)	84.87	# 6	Compare
Visual Question Answering (VQA)	IconQA	Patch-TRM	Sub-tasks (Img.)	82.66	# 1	Compare
			Sub-tasks (Txt.)	75.19	# 1	Compare
			Sub-tasks (Blank)	83.62	# 1	Compare
			Reasoning (Geo.)	81.87	# 3	Compare
			Reasoning (Cou.)	77.81	# 1	Compare
			Reasoning (Com.)	87.00	# 1	Compare
			Reasoning (Spa.)	55.62	# 1	Compare
			Reasoning (Sce.)	62.39	# 6	Compare
			Reasoning (Pat.)	68.75	# 1	Compare
			Reasoning (Tim.)	77.98	# 1	Compare
			Reasoning (Fra.)	82.13	# 1	Compare
			Reasoning (Est.)	98.24	# 8	Compare
			Reasoning (Alg.)	56.73	# 1	Compare
			Reasoning (Mea.)	97.98	# 8	Compare
			Reasoning (Sen.)	92.49	# 1	Compare
			Reasoning (Pro.)	95.73	# 1	Compare
Visual Question Answering (VQA)	IconQA	Random	Sub-tasks (Img.)	41.70	# 10	Compare
			Sub-tasks (Txt.)	36.87	# 10	Compare
			Sub-tasks (Blank)	0.29	# 12	Compare
			Reasoning (Geo.)	30.30	# 12	Compare
			Reasoning (Cou.)	18.38	# 12	Compare
			Reasoning (Com.)	41.20	# 12	Compare
			Reasoning (Spa.)	36.49	# 12	Compare
			Reasoning (Sce.)	34.25	# 12	Compare
			Reasoning (Pat.)	34.81	# 11	Compare
			Reasoning (Tim.)	35.82	# 12	Compare
			Reasoning (Fra.)	34.84	# 10	Compare
			Reasoning (Est.)	3.62	# 12	Compare
			Reasoning (Alg.)	11.12	# 12	Compare
			Reasoning (Mea.)	0.36	# 12	Compare
			Reasoning (Sen.)	45.16	# 12	Compare
			Reasoning (Pro.)	38.81	# 11	Compare

Methods

Add Remove

Absolute Position Encodings • Adam • BPE • Dense Connections • Dropout • Label Smoothing • Layer Normalization • Linear Layer • Multi-Head Attention • Position-Wise Feed-Forward Layer • Residual Connection • Scaled Dot-Product Attention • Softmax • Transformer

Edit Social Preview

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove