Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications... While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many types of transfer learning: fully sampled fine-tuning, linear probing, few-shot transfer and zero-shot transfer for novel images and objects. All of these properties are critical for our vision foundation model to serve general purpose vision tasks. Florence achieves new state-of-the-art results in majority of 44 representative benchmarks, e.g., ImageNet-1K zero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of 97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600. read more

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Cross-Modal Retrieval COCO 2014 Florence Image-to-text R@1 81.8 # 1
Image-to-text R@5 95.2 # 1
Text-to-image R@1 63.2 # 1
Text-to-image R@5 85.7 # 1
Object Detection COCO minival Florence-CoSwin-H box AP 62 # 1
Object Detection COCO test-dev Florence-CoSwin-H box AP 62.4 # 2
Image Classification ImageNet Florence-CoSwin-H Top 1 Accuracy 90.05% # 7
Top 5 Accuracy 99.02% # 1
Zero-Shot Transfer Image Classification ImageNet Florence-CoSwin-H (@384pix) Accuracy (Private) 83.7 # 3
Action Recognition In Videos Kinetics-400 Florence Top-1 Accuracy 86.5 # 1
Top-5 Accuracy 97.3 # 1
Action Recognition In Videos Kinetics-600 Florence Top-1 Accuracy 87.8 # 1
Top-5 Accuracy 97.8 # 1
Video Retrieval MSR-VTT-1kA Florence text-to-video R@1 37.6 # 6
text-to-video R@5 63.8 # 6
text-to-video R@10 72.6 # 6
Visual Question Answering VQA v2 test-dev Florence Accuracy 80.16 # 1
Visual Question Answering VQA v2 test-std Florence overall 80.36 # 1