Simple Open-Vocabulary Object Detection with Vision Transformers

Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Described Object Detection Description Detection Dataset OWL-ViT-base Intra-scenario FULL mAP 8.6 # 2
Intra-scenario PRES mAP 8.5 # 2
Intra-scenario ABS mAP 8.8 # 2
Open Vocabulary Object Detection LVIS v1.0 OWL-ViT (CLIP-L/14) AP novel-LVIS base training 25.6 # 9
AP novel-Unrestricted open-vocabulary training 31.2 # 2
One-Shot Object Detection MS COCO OWL-ViT (R50+H/32) AP 0.5 41.8 # 1

Methods