End-to-End Object Detection with Transformers

We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at https://github.com/facebookresearch/detr.

PDF Abstract ECCV 2020 PDF ECCV 2020 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Panoptic Segmentation COCO minival DETR-R101 (ResNet-101) PQ 45.1 # 22
SQ 79.9 # 4
RQ 55.5 # 3
PQth 50.5 # 19
SQth 80.9 # 4
RQth 61.7 # 2
PQst 37 # 16
SQst 78.5 # 2
RQst 46 # 2
AP 33 # 11
Panoptic Segmentation COCO minival PanopticFPN++ PQ 44.1 # 24
SQ 79.5 # 5
RQ 53.3 # 4
PQth 51.0 # 18
SQth 83.2 # 3
RQth 60.6 # 3
PQst 33.6 # 21
SQst 74.0 # 4
RQst 42.1 # 4
AP 39.7 # 10
Object Detection COCO minival DETR-DC5 (ResNet-101) box AP 44.9 # 117
AP50 64.7 # 46
AP75 47.7 # 49
APS 23.7 # 69
APM 49.5 # 27
APL 62.3 # 30
Object Detection COCO minival Faster RCNN-R101-FPN+ box AP 44 # 134
AP50 63.9 # 57
AP75 47.8 # 48
APS 27.2 # 34
APM 48.1 # 39
APL 56 # 66
Object Detection COCO-O DETR (ResNet-50) Average mAP 17.1 # 35
Effective Robustness -1.82 # 37
Real-Time Object Detection MS COCO Faster RCNN-FPN+ FPS (V100, b=1) 26 # 48
box AP 42 # 64

Methods