TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Multi-Object Tracking and Segmentation	BDD100K val	UNINEXT-H	mMOTSA	35.7	# 1
Multiple Object Tracking	BDD100K val	UNINEXT-H	mMOTA	44.2	# 2
Multiple Object Tracking	BDD100K val	UNINEXT-H	mIDF1	56.7	# 1
Object Detection	COCO minival	UNINEXT-H	box AP	60.6	# 19
Object Detection	COCO minival	UNINEXT-H	AP50	77.5	# 3
Object Detection	COCO minival	UNINEXT-H	AP75	66.7	# 2
Object Detection	COCO minival	UNINEXT-H	APS	45.1	# 2
Object Detection	COCO minival	UNINEXT-H	APM	64.8	# 2
Object Detection	COCO minival	UNINEXT-H	APL	75.3	# 2
Instance Segmentation	COCO test-dev	UNINEXT-H	mask AP	51.8	# 15
Instance Segmentation	COCO test-dev	UNINEXT-H	AP50	76.2	# 4
Instance Segmentation	COCO test-dev	UNINEXT-H	AP75	56.7	# 3
Instance Segmentation	COCO test-dev	UNINEXT-H	APS	33.3	# 5
Instance Segmentation	COCO test-dev	UNINEXT-H	APM	55.9	# 4
Instance Segmentation	COCO test-dev	UNINEXT-H	APL	67.5	# 5
Referring Expression Segmentation	DAVIS 2017 (val)	UNINEXT-H	J&F 1st frame	72.5	# 1
Described Object Detection	Description Detection Dataset	UNINEXT-large	Intra-scenario FULL mAP	17.9	# 5
Described Object Detection	Description Detection Dataset	UNINEXT-large	Intra-scenario PRES mAP	18.6	# 4
Described Object Detection	Description Detection Dataset	UNINEXT-large	Intra-scenario ABS mAP	15.9	# 4
Generalized Referring Expression Comprehension	gRefCOCO	UNINEXT	Precision@(F1=1, IoU≥0.5)	58.2	# 1
Generalized Referring Expression Comprehension	gRefCOCO	UNINEXT	N-acc.	50.6	# 1
Visual Object Tracking	LaSOT	UNINEXT-H	AUC	72.2	# 8
Visual Object Tracking	LaSOT	UNINEXT-H	Normalized Precision	80.8	# 8
Visual Object Tracking	LaSOT	UNINEXT-H	Precision	79.4	# 4
Visual Object Tracking	LaSOT	UNINEXT-L	AUC	72.4	# 7
Visual Object Tracking	LaSOT	UNINEXT-L	Normalized Precision	80.7	# 10
Visual Object Tracking	LaSOT	UNINEXT-L	Precision	78.9	# 6
Visual Object Tracking	LaSOT-ext	UNINEXT-H	AUC	56.2	# 1
Visual Object Tracking	LaSOT-ext	UNINEXT-H	Normalized Precision	63.8	# 1
Visual Object Tracking	LaSOT-ext	UNINEXT-H	Precision	63.8	# 1
Video Instance Segmentation	OVIS validation	UNINEXT (ResNet-50, Online)	mask AP	34.0	# 24
Video Instance Segmentation	OVIS validation	UNINEXT (ResNet-50, Online)	AP50	55.5	# 23
Video Instance Segmentation	OVIS validation	UNINEXT (ResNet-50, Online)	AP75	35.6	# 22
Video Instance Segmentation	OVIS validation	UNINEXT (ViT-H, Online)	mask AP	49.0	# 5
Video Instance Segmentation	OVIS validation	UNINEXT (ViT-H, Online)	AP50	72.5	# 3
Video Instance Segmentation	OVIS validation	UNINEXT (ViT-H, Online)	AP75	52.2	# 5
Referring Expression Comprehension	RefCoco+	UNINEXT-H	Val	85.24	# 3
Referring Expression Comprehension	RefCoco+	UNINEXT-H	Test A	89.63	# 4
Referring Expression Comprehension	RefCoco+	UNINEXT-H	Test B	79.79	# 3
Referring Expression Comprehension	RefCOCO	UNINEXT-H	Val	92.64	# 1
Referring Expression Comprehension	RefCOCO	UNINEXT-H	Test A	94.33	# 1
Referring Expression Comprehension	RefCOCO	UNINEXT-H	Test B	91.46	# 1
Referring Expression Comprehension	RefCOCOg-test	UNINEXT-H	Accuracy	89.37	# 1
Referring Expression Comprehension	RefCOCOg-val	UNINEXT-H	Accuracy	88.73	# 2
Referring Expression Segmentation	RefCOCO testA	UNINEXT-H	Overall IoU	83.44	# 1
Referring Expression Segmentation	RefCOCO+ testA	UNINEXT-H	Overall IoU	76.42	# 3
Referring Expression Segmentation	RefCOCO testB	UNINEXT-H	Overall IoU	81.33	# 1
Referring Expression Segmentation	RefCOCO+ test B	UNINEXT-H	Overall IoU	66.22	# 3
Referring Expression Segmentation	RefCoCo val	UNINEXT-H	Overall IoU	82.19	# 2
Referring Expression Segmentation	RefCoCo val	UNINEXT-H	Overall IoU	82.19	# 1
Referring Expression Segmentation	RefCOCO+ val	UNINEXT-H	Overall IoU	72.47	# 4
Referring Expression Segmentation	Refer-YouTube-VOS (2021 public validation)	UNINEXT-H	J&F	70.1	# 2
Referring Expression Segmentation	Refer-YouTube-VOS (2021 public validation)	UNINEXT-H	J	67.6	# 2
Referring Expression Segmentation	Refer-YouTube-VOS (2021 public validation)	UNINEXT-H	F	72.7	# 2
Zero Shot Segmentation	Segmentation in the Wild	UNINEXT	Mean AP	42.1	# 3
Visual Tracking	TNL2K	UNINEXT-H	precision	62.8	# 1
Visual Tracking	TNL2K	UNINEXT-H	AUC	59.3	# 2
Visual Object Tracking	TrackingNet	UNINEXT-H	Precision	86.4	# 1
Visual Object Tracking	TrackingNet	UNINEXT-H	Normalized Precision	89.0	# 6
Visual Object Tracking	TrackingNet	UNINEXT-H	Accuracy	85.4	# 6
Video Instance Segmentation	YouTube-VIS validation	UNINEXT-H	mask AP	66.9	# 4
Video Instance Segmentation	YouTube-VIS validation	UNINEXT-H	AP50	87.5	# 6
Video Instance Segmentation	YouTube-VIS validation	UNINEXT-H	AP75	75.1	# 3

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/multi-object-tracking-and-segmentation-on-3)](https://paperswithcode.com/sota/multi-object-tracking-and-segmentation-on-3?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-davis)](https://paperswithcode.com/sota/referring-expression-segmentation-on-davis?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/generalized-referring-expression)](https://paperswithcode.com/sota/generalized-referring-expression?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/visual-object-tracking-on-lasot-ext)](https://paperswithcode.com/sota/visual-object-tracking-on-lasot-ext?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-comprehension-on-refcoco)](https://paperswithcode.com/sota/referring-expression-comprehension-on-refcoco?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-comprehension-on-1)](https://paperswithcode.com/sota/referring-expression-comprehension-on-1?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco-1)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-1?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco-2)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-2?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco-7)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-7?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/multiple-object-tracking-on-bdd100k-val)](https://paperswithcode.com/sota/multiple-object-tracking-on-bdd100k-val?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-comprehension-on)](https://paperswithcode.com/sota/referring-expression-comprehension-on?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refer-1)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refer-1?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/visual-tracking-on-tnl2k)](https://paperswithcode.com/sota/visual-tracking-on-tnl2k?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco-4)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-4?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco-5)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-5?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/zero-shot-segmentation-on-segmentation-in-the)](https://paperswithcode.com/sota/zero-shot-segmentation-on-segmentation-in-the?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-comprehension-on-refcoco-1)](https://paperswithcode.com/sota/referring-expression-comprehension-on-refcoco-1?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/referring-expression-segmentation-on-refcoco-3)](https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-3?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/video-instance-segmentation-on-youtube-vis-1)](https://paperswithcode.com/sota/video-instance-segmentation-on-youtube-vis-1?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/described-object-detection-on-description)](https://paperswithcode.com/sota/described-object-detection-on-description?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/video-instance-segmentation-on-ovis-1)](https://paperswithcode.com/sota/video-instance-segmentation-on-ovis-1?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/visual-object-tracking-on-trackingnet)](https://paperswithcode.com/sota/visual-object-tracking-on-trackingnet?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/visual-object-tracking-on-lasot)](https://paperswithcode.com/sota/visual-object-tracking-on-lasot?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/instance-segmentation-on-coco)](https://paperswithcode.com/sota/instance-segmentation-on-coco?p=universal-instance-perception-as-object)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/universal-instance-perception-as-object/object-detection-on-coco-minival)](https://paperswithcode.com/sota/object-detection-on-coco-minival?p=universal-instance-perception-as-object)`

Universal Instance Perception as Object Discovery and Retrieval

CVPR 2023 · Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, Huchuan Lu ·

All instance perception tasks aim at finding certain objects specified by some queries such as category names, language expressions, and target annotations, but this complete field has been split into multiple independent subtasks. In this work, we present a universal instance perception model of the next generation, termed UNINEXT. UNINEXT reformulates diverse instance perception tasks into a unified object discovery and retrieval paradigm and can flexibly perceive different types of objects by simply changing the input prompts. This unified formulation brings the following benefits: (1) enormous data from different tasks and label vocabularies can be exploited for jointly training general instance-level representations, which is especially beneficial for tasks lacking in training data. (2) the unified model is parameter-efficient and can save redundant computation when handling multiple tasks simultaneously. UNINEXT shows superior performance on 20 challenging benchmarks from 10 instance-level tasks including classical image-level tasks (object detection and instance segmentation), vision-and-language tasks (referring expression comprehension and segmentation), and six video-level object tracking tasks. Code is available at https://github.com/MasterBin-IIAU/UNINEXT.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Code

Add Remove Mark official

MasterBin-IIAU/UNINEXT official

1,439

Tasks

Add Remove

Described Object Detection

Generalized Referring Expression Comprehension

Instance Segmentation

Multi-Object Tracking and Segmentation

Multiple Object Tracking

Object

object-detection

Object Detection

Object Discovery

Object Tracking

Referring Expression

Referring Expression Comprehension

Referring Expression Segmentation

Referring Video Object Segmentation

Retrieval

Semantic Segmentation

Video Instance Segmentation

Visual Object Tracking

Visual Tracking

Zero Shot Segmentation

Datasets

MS COCO

BDD100K

RefCOCO

DAVIS 2017

LaSOT

TrackingNet

YouTube-VOS 2018

YouTube-VIS 2019

Referring Expressions for DAVIS 2016 & 2017

OVIS

TNL2K

Refer-YouTube-VOS Google Refexp

gRefCOCO

Segmentation in the Wild

Description Detection Dataset

Results from the Paper

Edit

Ranked #1 on Referring Expression Segmentation on RefCoCo val (using extra training data)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Multi-Object Tracking and Segmentation	BDD100K val	UNINEXT-H	mMOTSA	35.7	# 1	Compare
Multiple Object Tracking	BDD100K val	UNINEXT-H	mMOTA	44.2	# 2	Compare
Multiple Object Tracking	BDD100K val	UNINEXT-H	mIDF1	56.7	# 1	Compare
Object Detection	COCO minival	UNINEXT-H	box AP	60.6	# 19	Compare
			AP50	77.5	# 3	Compare
			AP75	66.7	# 2	Compare
			APS	45.1	# 2	Compare
			APM	64.8	# 2	Compare
			APL	75.3	# 2	Compare
Instance Segmentation	COCO test-dev	UNINEXT-H	mask AP	51.8	# 15	Compare
			AP50	76.2	# 4	Compare
			AP75	56.7	# 3	Compare
			APS	33.3	# 5	Compare
			APM	55.9	# 4	Compare
			APL	67.5	# 5	Compare
Referring Expression Segmentation	DAVIS 2017 (val)	UNINEXT-H	J&F 1st frame	72.5	# 1	Compare
Described Object Detection	Description Detection Dataset	UNINEXT-large	Intra-scenario FULL mAP	17.9	# 5	Compare
			Intra-scenario PRES mAP	18.6	# 4	Compare
			Intra-scenario ABS mAP	15.9	# 4	Compare
Generalized Referring Expression Comprehension	gRefCOCO	UNINEXT	Precision@(F1=1, IoU≥0.5)	58.2	# 1	Compare
Generalized Referring Expression Comprehension	gRefCOCO	UNINEXT	N-acc.	50.6	# 1	Compare
Visual Object Tracking	LaSOT	UNINEXT-H	AUC	72.2	# 8	Compare
			Normalized Precision	80.8	# 8	Compare
			Precision	79.4	# 4	Compare
Visual Object Tracking	LaSOT	UNINEXT-L	AUC	72.4	# 7	Compare
			Normalized Precision	80.7	# 10	Compare
			Precision	78.9	# 6	Compare
Visual Object Tracking	LaSOT-ext	UNINEXT-H	AUC	56.2	# 1	Compare
			Normalized Precision	63.8	# 1	Compare
			Precision	63.8	# 1	Compare
Video Instance Segmentation	OVIS validation	UNINEXT (ResNet-50, Online)	mask AP	34.0	# 24	Compare
			AP50	55.5	# 23	Compare
			AP75	35.6	# 22	Compare
Video Instance Segmentation	OVIS validation	UNINEXT (ViT-H, Online)	mask AP	49.0	# 5	Compare
			AP50	72.5	# 3	Compare
			AP75	52.2	# 5	Compare
Referring Expression Comprehension	RefCoco+	UNINEXT-H	Val	85.24	# 3	Compare
			Test A	89.63	# 4	Compare
			Test B	79.79	# 3	Compare
Referring Expression Comprehension	RefCOCO	UNINEXT-H	Val	92.64	# 1	Compare
			Test A	94.33	# 1	Compare
			Test B	91.46	# 1	Compare
Referring Expression Comprehension	RefCOCOg-test	UNINEXT-H	Accuracy	89.37	# 1	Compare
Referring Expression Comprehension	RefCOCOg-val	UNINEXT-H	Accuracy	88.73	# 2	Compare
Referring Expression Segmentation	RefCOCO testA	UNINEXT-H	Overall IoU	83.44	# 1	Compare
Referring Expression Segmentation	RefCOCO+ testA	UNINEXT-H	Overall IoU	76.42	# 3	Compare
Referring Expression Segmentation	RefCOCO testB	UNINEXT-H	Overall IoU	81.33	# 1	Compare
Referring Expression Segmentation	RefCOCO+ test B	UNINEXT-H	Overall IoU	66.22	# 3	Compare
Referring Expression Segmentation	RefCoCo val	UNINEXT-H	Overall IoU	82.19	# 2	Compare
Referring Expression Segmentation	RefCoCo val	UNINEXT-H	Overall IoU	82.19	# 1	Compare
Referring Expression Segmentation	RefCOCO+ val	UNINEXT-H	Overall IoU	72.47	# 4	Compare
Referring Expression Segmentation	Refer-YouTube-VOS (2021 public validation)	UNINEXT-H	J&F	70.1	# 2	Compare
			J	67.6	# 2	Compare
			F	72.7	# 2	Compare
Zero Shot Segmentation	Segmentation in the Wild	UNINEXT	Mean AP	42.1	# 3	Compare
Visual Tracking	TNL2K	UNINEXT-H	precision	62.8	# 1	Compare
Visual Tracking	TNL2K	UNINEXT-H	AUC	59.3	# 2	Compare
Visual Object Tracking	TrackingNet	UNINEXT-H	Precision	86.4	# 1	Compare
			Normalized Precision	89.0	# 6	Compare
			Accuracy	85.4	# 6	Compare
Video Instance Segmentation	YouTube-VIS validation	UNINEXT-H	mask AP	66.9	# 4	Compare
			AP50	87.5	# 6	Compare
			AP75	75.1	# 3	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

Universal Instance Perception as Object Discovery and Retrieval

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove