OneFormer: One Transformer to Rule Universal Image Segmentation

10 Nov 2022  ·  Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi ·

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible. To support further research, we open-source our code and models at https://github.com/SHI-Labs/OneFormer

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Instance Segmentation ADE20K val OneFormer (Swin-L, single-scale) AP 35.9 # 4
Panoptic Segmentation ADE20K val OneFormer (DiNAT-L, single-scale, 640x640) PQ 50.5 # 6
AP 36.0 # 7
mIoU 58.3 # 3
Panoptic Segmentation ADE20K val OneFormer (ConvNeXt-L, single-scale, 640x640) PQ 50.0 # 8
AP 36.2 # 6
mIoU 56.6 # 9
Panoptic Segmentation ADE20K val OneFormer (Swin-L, single-scale, 640x640) PQ 49.8 # 9
AP 35.9 # 8
mIoU 57.0 # 7
Panoptic Segmentation ADE20K val OneFormer (DiNAT-L, single-scale, 1280x1280, COCO-Pretrain) PQ 53.4 # 1
AP 40.2 # 1
mIoU 58.9 # 2
Semantic Segmentation ADE20K val OneFormer (Swin-L, multi-scale, 896x896) mIoU 58.3 # 9
Semantic Segmentation ADE20K val OneFormer (DiNAT-L, single-scale, 640x640) mIoU 58.3 # 9
Instance Segmentation ADE20K val OneFormer (DiNAT-L, single-scale, 1280x1280, COCO-pretrain) AP 40.2 # 1
APS 19.2 # 1
APM 44.4 # 1
APL 59.7 # 1
Semantic Segmentation ADE20K val OneFormer (DiNAT-L, multi-scale, 640x640) mIoU 58.4 # 6
Panoptic Segmentation ADE20K val OneFormer (DiNAT-L, single-scale, 1280x1280) PQ 51.5 # 3
AP 37.1 # 4
mIoU 58.3 # 3
Panoptic Segmentation ADE20K val OneFormer (Swin-L, single-scale, 1280x1280) PQ 51.4 # 4
AP 37.8 # 3
mIoU 57.0 # 7
Instance Segmentation ADE20K val OneFormer (DiNAT-L, single-scale) AP 36.0 # 3
Panoptic Segmentation ADE20K val OneFormer (ConvNeXt-XL, single-scale, 640x640) PQ 50.1 # 7
AP 36.3 # 5
mIoU 57.4 # 6
Semantic Segmentation ADE20K val OneFormer (DiNAT-L, multi-scale, 896x896) mIoU 58.6 # 5
Semantic Segmentation ADE20K val OneFormer (Swin-L, multi-scale, 640x640) mIoU 57.7 # 14
Panoptic Segmentation Cityscapes test OneFormer (ConvNeXt-L, single-scale, Mapillary Vistas-Pretrained) PQ 68.0 # 1
Panoptic Segmentation Cityscapes val OneFormer (ConvNeXt-L, single-scale) PQ 68.51 # 3
mIoU 83.0 # 10
AP 46.5 # 4
Panoptic Segmentation Cityscapes val OneFormer (ConvNeXt-L, single-scale, 512x1024, Mapillary Vistas-pretrained) PQ 70.1 # 1
PQst 74.1 # 1
PQth 64.6 # 1
mIoU 84.6 # 3
AP 48.7 # 1
Instance Segmentation Cityscapes val OneFormer (Swin-L, single-scale) mask AP 45.6 # 1
Instance Segmentation Cityscapes val OneFormer (DiNAT-L, single-scale) mask AP 45.6 # 1
Semantic Segmentation Cityscapes val OneFormer (Swin-L, multi-scale) mIoU 84.4 # 9
Panoptic Segmentation Cityscapes val OneFormer (Swin-L, single-scale) PQ 67.2 # 10
mIoU 83.0 # 10
AP 45.6 # 5
Panoptic Segmentation Cityscapes val OneFormer (DiNAT-L, single-scale) PQ 67.6 # 8
mIoU 83.1 # 9
AP 45.6 # 5
Semantic Segmentation Cityscapes val OneFormer (ConvNeXt-XL, multi-scale) mIoU 84.6 # 7
Panoptic Segmentation Cityscapes val OneFormer (ConvNeXt-XL, single-scale) PQ 68.4 # 6
mIoU 83.6 # 6
AP 46.7 # 3
Semantic Segmentation Cityscapes val OneFormer (ConvNeXt-L, Mapillary, multi-scale) mIoU 85.2 # 4
Semantic Segmentation COCO OneFormer (DiNAT-L, single-scale) mIoU 68.1 # 1
Semantic Segmentation COCO OneFormer (Swin-L, single-scale) mIoU 67.4 # 2
Panoptic Segmentation COCO minival OneFormer (Swin-L, single-scale) PQ 57.9 # 8
PQth 64.4 # 3
PQst 48.0 # 8
AP 49.0 # 5
mIoU 67.4 # 3
Panoptic Segmentation COCO minival OneFormer (DiNAT-L, single-scale) PQ 58.0 # 6
PQth 64.3 # 4
PQst 48.4 # 5
AP 49.2 # 3
mIoU 68.1 # 2
Instance Segmentation COCO val (panoptic labels) OneFormer (DiNAT-L, single-scale) AP 49.2 # 1
Instance Segmentation COCO val (panoptic labels) OneFormer (Swin-L, single-scale) AP 49.0 # 3
Semantic Segmentation Mapillary val OneFormer (DiNAT-L, multi-scale) mIoU 64.9 # 1
Panoptic Segmentation Mapillary val OneFormer (DiNAT-L, single-scale) PQ 46.7 # 1
mIoU 61.7 # 1
PQth 40.5 # 3
PQst 54.9 # 1
Panoptic Segmentation Mapillary val OneFormer (ConvNeXt-L, single-scale) PQ 46.4 # 2
mIoU 61.6 # 2
PQth 40.6 # 2
PQst 54.0 # 2

Methods