USB: Universal-Scale Object Detection Benchmark

25 Mar 2021  ·  Yosuke Shinya ·

Benchmarks, such as COCO, play a crucial role in object detection. However, existing benchmarks are insufficient in scale variation, and their protocols are inadequate for fair comparison. In this paper, we introduce the Universal-Scale object detection Benchmark (USB). USB has variations in object scales and image domains by incorporating COCO with the recently proposed Waymo Open Dataset and Manga109-s dataset. To enable fair comparison and inclusive research, we propose training and evaluation protocols. They have multiple divisions for training epochs and evaluation image resolutions, like weight classes in sports, and compatibility across training protocols, like the backward compatibility of the Universal Serial Bus. Specifically, we request participants to report results with not only higher protocols (longer training) but also lower protocols (shorter training). Using the proposed benchmark and protocols, we analyzed eight methods and found weaknesses of existing COCO-biased methods. The code is available at https://github.com/shinya7y/UniverseNet .

PDF Abstract

Datasets


Results from the Paper


 Ranked #1 on Object Detection on Waymo 2D detection all_ns test (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Object Detection COCO minival UniverseNet-20.08d (Res2Net-101, DCN, single-scale) box AP 50.9 # 35
AP50 69.5 # 15
AP75 55.4 # 10
APS 33.5 # 11
APM 55.5 # 11
APL 65.8 # 12
Object Detection COCO minival UniverseNet-20.08 (Res2Net-50, DCN, single-scale) box AP 48.5 # 46
AP50 67.0 # 21
AP75 52.6 # 18
APS 30.6 # 13
APM 52.7 # 14
APL 62.7 # 14
Object Detection COCO minival UniverseNet-20.08d (Res2Net-101, DCN, multi-scale) box AP 53.5 # 25
AP50 70.8 # 10
AP75 58.9 # 6
APS 36.9 # 7
APM 57.5 # 7
APL 68.1 # 8
Object Detection COCO test-dev UniverseNet-20.08 (Res2Net-50, DCN, single-scale) box AP 48.8 # 69
AP50 67.5 # 56
AP75 53.0 # 50
APS 30.1 # 52
APM 52.3 # 47
APL 61.1 # 52
Hardware Burden None # 1
Operations per network pass None # 1
Object Detection COCO test-dev UniverseNet-20.08d (Res2Net-101, DCN, multi-scale) box AP 54.1 # 32
AP50 71.6 # 24
AP75 59.9 # 17
APS 35.8 # 16
APM 57.2 # 18
APL 67.4 # 12
Hardware Burden None # 1
Operations per network pass None # 1
Object Detection COCO test-dev UniverseNet-20.08d (Res2Net-101, DCN, single-scale) box AP 51.3 # 51
AP50 70.0 # 35
AP75 55.8 # 35
APS 31.7 # 41
APM 55.3 # 29
APL 64.9 # 28
Hardware Burden None # 1
Operations per network pass None # 1
Object Detection Manga109-s 15test RetinaNet COCO-style AP 65.3 # 8
Object Detection Manga109-s 15test ATSS COCO-style AP 66.5 # 6
Object Detection Manga109-s 15test ATSEPC COCO-style AP 67.1 # 5
Object Detection Manga109-s 15test GFL COCO-style AP 67.3 # 4
Object Detection Manga109-s 15test UniverseNet-20.08 COCO-style AP 69.9 # 1
Object Detection Manga109-s 15test UniverseNet COCO-style AP 68.9 # 2
Object Detection Manga109-s 15test Faster R-CNN COCO-style AP 65.8 # 7
Object Detection Manga109-s 15test Cascade R-CNN COCO-style AP 67.6 # 3
Object Detection USB (Standard USB 1.0 protocol) RetinaNet mCAP 44.8 # 8
Object Detection USB (Standard USB 1.0 protocol) ATSEPC mCAP 48.1 # 3
Object Detection USB (Standard USB 1.0 protocol) GFL mCAP 47.7 # 5
Object Detection USB (Standard USB 1.0 protocol) UniverseNet mCAP 51.4 # 2
Object Detection USB (Standard USB 1.0 protocol) UniverseNet-20.08 mCAP 52.1 # 1
Object Detection USB (Standard USB 1.0 protocol) Faster R-CNN mCAP 45.9 # 7
Object Detection USB (Standard USB 1.0 protocol) Cascade R-CNN mCAP 48.1 # 3
Object Detection USB (Standard USB 1.0 protocol) ATSS mCAP 47.1 # 6
Object Detection Waymo 2D detection all_ns f0val RetinaNet COCO-style AP 32.5 # 8
Object Detection Waymo 2D detection all_ns f0val UniverseNet COCO-style AP 38.6 # 2
Object Detection Waymo 2D detection all_ns f0val ATSS COCO-style AP 35.4 # 5
Object Detection Waymo 2D detection all_ns f0val ATSEPC COCO-style AP 35.0 # 6
Object Detection Waymo 2D detection all_ns f0val Faster R-CNN COCO-style AP 34.5 # 7
Object Detection Waymo 2D detection all_ns f0val UniverseNet-20.08 COCO-style AP 39.0 # 1
Object Detection Waymo 2D detection all_ns f0val GFL COCO-style AP 35.7 # 4
Object Detection Waymo 2D detection all_ns f0val Cascade R-CNN COCO-style AP 36.4 # 3
Object Detection Waymo 2D detection all_ns test UniverseNet AP/L2 67.42 # 1

Methods


No methods listed for this paper. Add relevant methods here