DiffusionInst: Diffusion Model for Instance Segmentation

6 Dec 2022  ·  Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, Weiqiang Wang ·

Diffusion frameworks have achieved comparable performance with previous state-of-the-art image generation models. Researchers are curious about its variants in discriminative tasks because of its powerful noise-to-image denoising pipeline. This paper proposes DiffusionInst, a novel framework that represents instances as instance-aware filters and formulates instance segmentation as a noise-to-filter denoising process. The model is trained to reverse the noisy groundtruth without any inductive bias from RPN. During inference, it takes a randomly generated filter as input and outputs mask in one-step or multi-step denoising. Extensive experimental results on COCO and LVIS show that DiffusionInst achieves competitive performance compared to existing instance segmentation models with various backbones, such as ResNet and Swin Transformers. We hope our work could serve as a strong baseline, which could inspire designing more efficient diffusion frameworks for challenging discriminative tasks. Our code is available in https://github.com/chenhaoxing/DiffusionInst.

PDF Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Instance Segmentation COCO test-dev DiffusionInst-ResNet50 mask AP 37.1 # 77
Instance Segmentation COCO test-dev DiffusionInst-SwinL mask AP 48.3 # 21
Instance Segmentation COCO test-dev DiffusionInst-SwinB mask AP 47.6 # 23
Instance Segmentation COCO test-dev DiffusionInst-ResNet101 mask AP 41.5 # 47
Instance Segmentation LVIS v1.0 val DiffusionInst-SwinL mask AP 38.6 # 5
Instance Segmentation LVIS v1.0 val DiffusionInst-SwinB mask AP 36 # 7
Instance Segmentation LVIS v1.0 val DiffusionInst-ResNet101 mask AP 27 # 13
Instance Segmentation LVIS v1.0 val DiffusionInst-ResNet50 mask AP 22.3 # 16