SOLQ: Segmenting Objects by Learning Queries

In this paper, we propose an end-to-end framework for instance segmentation. Based on the recently introduced DETR [1], our method, termed SOLQ, segments objects by learning unified queries. In SOLQ, each query represents one object and has multiple representations: class, location and mask. The object queries learned perform classification, box regression and mask encoding simultaneously in an unified vector form. During training phase, the mask vectors encoded are supervised by the compression coding of raw spatial masks. In inference time, mask vectors produced can be directly transformed to spatial masks by the inverse process of compression coding. Experimental results show that SOLQ can achieve state-of-the-art performance, surpassing most of existing approaches. Moreover, the joint learning of unified query representation can greatly improve the detection performance of DETR. We hope our SOLQ can serve as a strong baseline for the Transformer-based instance segmentation. Code is available at https://github.com/megvii-research/SOLQ.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Instance Segmentation COCO minival SOLQ (Swin-L, single scale) mask AP 46.3 # 17
Object Detection COCO minival SOLQ (Swin-L, single scale) box AP 56.2 # 17
AP50 74.9 # 5
AP75 61.3 # 4
APS 38.8 # 6
APM 60.3 # 4
APL 71.9 # 4
Instance Segmentation COCO minival SOLQ (ResNet101, single scale) mask AP 40.4 # 43
Instance Segmentation COCO test-dev SOLQ (ResNet50, single scale) mask AP 39.7 # 39
Object Detection COCO test-dev SOLQ (Swin-L, single) box AP 56.5 # 18
Hardware Burden None # 1
Operations per network pass None # 1
Object Detection COCO test-dev SOLQ (ResNet101, single scale) box AP 48.7 # 70
Hardware Burden None # 1
Operations per network pass None # 1
Instance Segmentation COCO test-dev SOLQ (Swin-L, single scale) mask AP 46.7 # 15
Object Detection COCO test-dev SOLQ (ResNet50, single scale) box AP 47.8 # 78
Hardware Burden None # 1
Operations per network pass None # 1
Instance Segmentation COCO test-dev SOLQ (ResNet101, single scale) mask AP 40.9 # 29

Methods