USB: Universal-Scale Object Detection Benchmark

25 Mar 2021  ·  Yosuke Shinya ·

Benchmarks, such as COCO, play a crucial role in object detection. However, existing benchmarks are insufficient in scale variation, and their protocols are inadequate for fair comparison. In this paper, we introduce the Universal-Scale object detection Benchmark (USB). USB has variations in object scales and image domains by incorporating COCO with the recently proposed Waymo Open Dataset and Manga109-s dataset. To enable fair comparison and inclusive research, we propose training and evaluation protocols. They have multiple divisions for training epochs and evaluation image resolutions, like weight classes in sports, and compatibility across training protocols, like the backward compatibility of the Universal Serial Bus. Specifically, we request participants to report results with not only higher protocols (longer training) but also lower protocols (shorter training). Using the proposed benchmark and protocols, we conducted extensive experiments using 15 methods and found weaknesses of existing COCO-biased methods. The code is available at https://github.com/shinya7y/UniverseNet .

PDF Abstract

Datasets


Results from the Paper


 Ranked #1 on Object Detection on Manga109-s 15test (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Object Detection COCO minival UniverseNet-20.08d (Res2Net-101, DCN, multi-scale) box AP 53.5 # 58
AP50 70.8 # 16
AP75 58.9 # 9
APS 36.9 # 10
APM 57.5 # 9
APL 68.1 # 12
Object Detection COCO minival UniverseNet-20.08d (Res2Net-101, DCN, single-scale) box AP 50.9 # 74
AP50 69.5 # 24
AP75 55.4 # 18
APS 33.5 # 16
APM 55.5 # 14
APL 65.8 # 18
Object Detection COCO minival UniverseNet-20.08 (Res2Net-50, DCN, single-scale) box AP 48.5 # 87
AP50 67.0 # 34
AP75 52.6 # 28
APS 30.6 # 19
APM 52.7 # 19
APL 62.7 # 29
Object Detection COCO-O UniverseNet (R2-101-DCN) Average mAP 24.8 # 28
Object Detection COCO-O UniverseNet (R2-101-DCN) Effective Robustness 1.86 # 29
Object Detection COCO test-dev UniverseNet-20.08d (Res2Net-101, DCN, multi-scale) box mAP 54.1 # 54
AP50 71.6 # 25
AP75 59.9 # 18
APS 35.8 # 15
APM 57.2 # 17
APL 67.4 # 14
Object Detection COCO test-dev UniverseNet-20.08 (Res2Net-50, DCN, single-scale) box mAP 48.8 # 99
AP50 67.5 # 58
AP75 53.0 # 53
APS 30.1 # 51
APM 52.3 # 45
APL 61.1 # 51
Object Detection COCO test-dev UniverseNet-20.08d (Res2Net-101, DCN, single-scale) box mAP 51.3 # 78
AP50 70.0 # 37
AP75 55.8 # 38
APS 31.7 # 40
APM 55.3 # 27
APL 64.9 # 27
Object Detection Manga109-s 15test Sparse R-CNN COCO-style AP 63.1 # 14
Object Detection Manga109-s 15test ATSS (ConvNeXt-T) COCO-style AP 67.4 # 6
Object Detection Manga109-s 15test ATSS+DyHead COCO-style AP 67.9 # 4
Object Detection Manga109-s 15test YOLOX-L COCO-style AP 70.2 # 1
Object Detection Manga109-s 15test Deformable DETR COCO-style AP 64.1 # 13
Object Detection Manga109-s 15test DETR COCO-style AP 31.2 # 15
Object Detection Manga109-s 15test ATSS (Swin-T) COCO-style AP 66.2 # 10
Object Detection Manga109-s 15test UniverseNet COCO-style AP 68.9 # 3
Object Detection Manga109-s 15test GFL COCO-style AP 67.3 # 7
Object Detection Manga109-s 15test ATSS+SEPC COCO-style AP 67.1 # 8
Object Detection Manga109-s 15test ATSS COCO-style AP 66.5 # 9
Object Detection Manga109-s 15test RetinaNet COCO-style AP 65.3 # 12
Object Detection Manga109-s 15test Cascade R-CNN COCO-style AP 67.6 # 5
Object Detection Manga109-s 15test Faster R-CNN COCO-style AP 65.8 # 11
Object Detection Manga109-s 15test UniverseNet-20.08 COCO-style AP 69.9 # 2
Object Detection USB (Standard USB 1.0 protocol) ATSS (Swin-T) mCAP 49.0 # 6
Object Detection USB (Standard USB 1.0 protocol) ATSS+DyHead mCAP 49.4 # 5
Object Detection USB (Standard USB 1.0 protocol) UniverseNet-20.08 mCAP 52.1 # 1
Object Detection USB (Standard USB 1.0 protocol) UniverseNet mCAP 51.4 # 2
Object Detection USB (Standard USB 1.0 protocol) GFL mCAP 47.7 # 8
Object Detection USB (Standard USB 1.0 protocol) ATSS mCAP 47.1 # 9
Object Detection USB (Standard USB 1.0 protocol) RetinaNet mCAP 44.8 # 11
Object Detection USB (Standard USB 1.0 protocol) Cascade R-CNN mCAP 48.1 # 7
Object Detection USB (Standard USB 1.0 protocol) Faster R-CNN mCAP 45.9 # 10
Object Detection USB (Standard USB 1.0 protocol) YOLOX-L mCAP 49.6 # 4
Object Detection USB (Standard USB 1.0 protocol) Sparse R-CNN mCAP 44.6 # 12
Object Detection USB (Standard USB 1.0 protocol) DETR mCAP 23.7 # 13
Object Detection USB (Standard USB 1.0 protocol) ATSS (ConvNeXt-T) mCAP 50.4 # 3
Object Detection Waymo 2D detection all_ns f0val GFL COCO-style AP 35.7 # 8
Object Detection Waymo 2D detection all_ns f0val Cascade R-CNN COCO-style AP 36.4 # 7
Object Detection Waymo 2D detection all_ns f0val Faster R-CNN COCO-style AP 34.5 # 11
Object Detection Waymo 2D detection all_ns f0val UniverseNet-20.08 COCO-style AP 39.0 # 2
Object Detection Waymo 2D detection all_ns f0val Deformable DETR COCO-style AP 32.7 # 13
Object Detection Waymo 2D detection all_ns f0val Sparse R-CNN COCO-style AP 32.8 # 12
Object Detection Waymo 2D detection all_ns f0val RetinaNet COCO-style AP 32.5 # 14
Object Detection Waymo 2D detection all_ns f0val ATSS+SEPC COCO-style AP 35.0 # 10
Object Detection Waymo 2D detection all_ns f0val ATSS COCO-style AP 35.4 # 9
Object Detection Waymo 2D detection all_ns f0val ATSS (ConvNeXt-T) COCO-style AP 38.3 # 4
Object Detection Waymo 2D detection all_ns f0val ATSS+DyHead COCO-style AP 37.1 # 6
Object Detection Waymo 2D detection all_ns f0val YOLOX-L COCO-style AP 41.6 # 1
Object Detection Waymo 2D detection all_ns f0val ATSS (Swin-T) COCO-style AP 37.2 # 5
Object Detection Waymo 2D detection all_ns f0val DETR COCO-style AP 17.8 # 15
Object Detection Waymo 2D detection all_ns f0val UniverseNet COCO-style AP 38.6 # 3
Object Detection Waymo 2D detection all_ns test UniverseNet AP/L2 67.42 # 1

Methods