TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Image Captioning	COCO Captions	OFA	BLEU-4	44.9	# 2
Image Captioning	COCO Captions	OFA	METEOR	32.5	# 3
Image Captioning	COCO Captions	OFA	CIDER	154.9	# 2
Image Captioning	COCO Captions	OFA	SPICE	26.6	# 2
Text Summarization	GigaWord	OFA	ROUGE-1	39.81	# 4
Text Summarization	GigaWord	OFA	ROUGE-2	20.66	# 3
Text Summarization	GigaWord	OFA	ROUGE-L	37.11	# 2
Object Categorization	GRIT	OFA_Large	Categorization (ablation)	22.6	# 4
Visual Question Answering	GRIT	OFA	VQA (ablation)	72.4	# 1
Referring Expression Comprehension	GRIT	OFA	Refexp (ablation)	61.7	# 2
Self-Supervised Image Classification	ImageNet (finetuned)	OFA (Large)	Number of Params	473M	# 11
Self-Supervised Image Classification	ImageNet (finetuned)	OFA (Large)	Top 1 Accuracy	85.6%	# 22
Referring Expression Comprehension	RefCoco+	OFA	Val	87.86	# 2
Referring Expression Comprehension	RefCoco+	OFA	Test A	91.70	# 2
Referring Expression Comprehension	RefCoco+	OFA	Test B	80.71	# 2
Referring Expression Comprehension	RefCOCO	OFA	Val	92.04	# 3
Referring Expression Comprehension	RefCOCO	OFA	Test A	94.03	# 3
Referring Expression Comprehension	RefCOCO	OFA	Test B	88.44	# 3
Referring Expression Comprehension	RefCOCOg-test	OFA	Accuracy	88.78	# 3
Referring Expression Comprehension	RefCOCOg-val	OFA	Accuracy	88.07	# 3
Visual Entailment	SNLI-VE test	OFA	Accuracy	91.2	# 1
Visual Entailment	SNLI-VE val	OFA	Accuracy	91.0	# 1
Visual Question Answering	VQA v2 test-dev	OFA	Accuracy	82.0	# 3
Visual Question Answering	VQA v2 test-std	OFA	yes/no	94.66	# 1
Visual Question Answering	VQA v2 test-std	OFA	number	71.44	# 1
Visual Question Answering	VQA v2 test-std	OFA	other	73.35	# 1
Visual Question Answering	VQA v2 test-std	OFA	overall	81.98	# 1

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/visual-question-answering-on-grit-1)](https://paperswithcode.com/sota/visual-question-answering-on-grit-1?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/visual-entailment-on-snli-ve-test)](https://paperswithcode.com/sota/visual-entailment-on-snli-ve-test?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/visual-entailment-on-snli-ve-val)](https://paperswithcode.com/sota/visual-entailment-on-snli-ve-val?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/visual-question-answering-on-vqa-v2-test-std-1)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-std-1?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/image-captioning-on-coco-captions)](https://paperswithcode.com/sota/image-captioning-on-coco-captions?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/referring-expression-comprehension-on-grit)](https://paperswithcode.com/sota/referring-expression-comprehension-on-grit?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/referring-expression-comprehension-on-refcoco-1)](https://paperswithcode.com/sota/referring-expression-comprehension-on-refcoco-1?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/referring-expression-comprehension-on-refcoco)](https://paperswithcode.com/sota/referring-expression-comprehension-on-refcoco?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/referring-expression-comprehension-on-1)](https://paperswithcode.com/sota/referring-expression-comprehension-on-1?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/referring-expression-comprehension-on)](https://paperswithcode.com/sota/referring-expression-comprehension-on?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/visual-question-answering-on-vqa-v2-test-dev-1)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-dev-1?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/text-summarization-on-gigaword)](https://paperswithcode.com/sota/text-summarization-on-gigaword?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/object-categorization-on-grit)](https://paperswithcode.com/sota/object-categorization-on-grit?p=unifying-architectures-tasks-and-modalities)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/unifying-architectures-tasks-and-modalities/self-supervised-image-classification-on-1)](https://paperswithcode.com/sota/self-supervised-image-classification-on-1?p=unifying-architectures-tasks-and-modalities)`

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

7 Feb 2022 · Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang ·

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.

PDF Abstract

Code

Add Remove Mark official

ofa-sys/ofa official

2,321

modelscope/modelscope

6,039

JHKim-snu/GVCCI

JHKim-snu/PGA

Tasks

Add Remove

Image Captioning

Image Classification

Image Generation

Language Modelling

Object Categorization

Referring Expression

Referring Expression Comprehension

Self-Supervised Image Classification

Text Generation

Text Summarization

Text-to-Image Generation

Visual Entailment

Visual Grounding

Visual Question Answering

Visual Question Answering (VQA)

Visual Reasoning

Datasets

ImageNet

Visual Question Answering v2.0

RefCOCO

COCO Captions SNLI-VE Google Refexp

GRIT

Results from the Paper

Edit

Ranked #1 on Visual Question Answering on VQA v2 test-std (yes/no metric)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Image Captioning	COCO Captions	OFA	BLEU-4	44.9	# 2	Compare
			METEOR	32.5	# 3	Compare
			CIDER	154.9	# 2	Compare
			SPICE	26.6	# 2	Compare
Text Summarization	GigaWord	OFA	ROUGE-1	39.81	# 4	Compare
			ROUGE-2	20.66	# 3	Compare
			ROUGE-L	37.11	# 2	Compare
Object Categorization	GRIT	OFA_Large	Categorization (ablation)	22.6	# 4	Compare
Visual Question Answering	GRIT	OFA	VQA (ablation)	72.4	# 1	Compare
Referring Expression Comprehension	GRIT	OFA	Refexp (ablation)	61.7	# 2	Compare
Self-Supervised Image Classification	ImageNet (finetuned)	OFA (Large)	Number of Params	473M	# 11	Compare
Self-Supervised Image Classification	ImageNet (finetuned)	OFA (Large)	Top 1 Accuracy	85.6%	# 22	Compare
Referring Expression Comprehension	RefCoco+	OFA	Val	87.86	# 2	Compare
			Test A	91.70	# 2	Compare
			Test B	80.71	# 2	Compare
Referring Expression Comprehension	RefCOCO	OFA	Val	92.04	# 3	Compare
			Test A	94.03	# 3	Compare
			Test B	88.44	# 3	Compare
Referring Expression Comprehension	RefCOCOg-test	OFA	Accuracy	88.78	# 3	Compare
Referring Expression Comprehension	RefCOCOg-val	OFA	Accuracy	88.07	# 3	Compare
Visual Entailment	SNLI-VE test	OFA	Accuracy	91.2	# 1	Compare
Visual Entailment	SNLI-VE val	OFA	Accuracy	91.0	# 1	Compare
Visual Question Answering	VQA v2 test-dev	OFA	Accuracy	82.0	# 3	Compare
Visual Question Answering	VQA v2 test-std	OFA	yes/no	94.66	# 1	Compare
			number	71.44	# 1	Compare
			other	73.35	# 1	Compare
			overall	81.98	# 1	Compare

Methods

Add Remove

1x1 Convolution • Adam • Attention Dropout • Average Pooling • Batch Normalization • BERT • Bottleneck Residual Block • ColorJitter • Convolution • Dense Connections • Dropout • Feedforward Network • GELU • Global Average Pooling • InfoNCE • Kaiming Initialization • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • Max Pooling • MoCo • MoCo v3 • Multi-Head Attention • NT-Xent • OFA • Random Gaussian Blur • Random Resized Crop • ReLU • Residual Block • Residual Connection • ResNet • Scaled Dot-Product Attention • SimCLR • Softmax • Weight Decay • WordPiece

Edit Social Preview

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove