Scaling Open-Vocabulary Object Detection

Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.

PDF Abstract NeurIPS 2023 PDF NeurIPS 2023 Abstract

Results from the Paper

Ranked #2 on Zero-Shot Object Detection on LVIS v1.0 val (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Zero-Shot Object Detection LVIS v1.0 minival OWLv2 (OWL-ST+FT) AP 51.3 # 2
Zero-Shot Object Detection LVIS v1.0 val OWLv2 (OWL-ST+FT) AP 47.0 # 2


No methods listed for this paper. Add relevant methods here