Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation

In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Some key components in DINO are extended for segmentation through a shared architecture and training process. Mask DINO is simple, efficient, and scalable, and it can benefit from joint large-scale detection and segmentation datasets. Our experiments show that Mask DINO significantly outperforms all existing specialized segmentation methods, both on a ResNet-50 backbone and a pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K) among models under one billion parameters. Code is available at \url{}.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper

 Ranked #1 on Panoptic Segmentation on COCO test-dev (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Semantic Segmentation ADE20K MasK DINO (SwinL, multi-scale) Validation mIoU 60.8 # 9
Params (M) 223 # 17
Semantic Segmentation ADE20K val MaskDINO-SwinL mIoU 60.8 # 4
Panoptic Segmentation COCO minival MasK DINO (SwinL,single-scale) PQ 59.4 # 3
AP 50.9 # 3
Instance Segmentation COCO minival MasK DINO (SwinL, multi-scale) mask AP 54.5 # 4
Instance Segmentation COCO minival Mask DINO (SwinL) mask AP 52.6 # 8
Instance Segmentation COCO test-dev Mask DINO (SwinL, single -scale) mask AP 52.8 # 10
Instance Segmentation COCO test-dev MasK DINO (SwinL, multi-scale) mask AP 54.7 # 5
Panoptic Segmentation COCO test-dev Mask DINO (single scale) PQ 59.5 # 1
PQst - # 34
PQth - # 35