DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.
PDF AbstractCode
CIFAR-10
ImageNet
Cityscapes
KITTI
UCF101
Kinetics
ADE20K
NYUv2
Places205
ImageNet-C
iNaturalist
SUN RGB-D
Oxford-IIIT Pet Dataset
Oxford RobotCar Dataset
Met
AmsterTime
Results from the Paper
Ranked #1 on
Image Classification
on CIFAR-10
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Semantic Segmentation | ADE20K | DINOv2 (ViT-g/14 frozen model, w/ ViT-Adapter + Mask2former) | Validation mIoU | 60.2 | # 11 | ||
| Params (M) | 1100 | # 6 | |||||
| Image Retrieval | AmsterTime | DINOv2 distilled (ViT-B/14 frozen) | mAP | 45.6 | # 3 | ||
| Image Retrieval | AmsterTime | DINOv2 distilled (ViT-L/14 frozen) | mAP | 50.0 | # 1 | ||
| Image Retrieval | AmsterTime | DINOv2 (ViT-g/14 frozen) | mAP | 46.7 | # 2 | ||
| Image Retrieval | AmsterTime | DINOv2 distilled (ViT-S/14 frozen) | mAP | 43.5 | # 4 | ||
| Image Classification | CIFAR-10 | DINOv2 (ViT-g/14, frozen model, linear eval) | Percentage correct | 99.5 | # 1 | ||
| PARAMS | 1100M | # 234 | |||||
| Self-Supervised Image Classification | ImageNet | DINOv2 (ViT-g/14 @448) | Top 1 Accuracy | 86.7% | # 1 | ||
| Number of Params | 1100M | # 2 | |||||
| Top 1 Accuracy (kNN, k=20) | 83.7% | # 1 | |||||
| Self-Supervised Image Classification | ImageNet | DINOv2 distilled (ViT-B/14) | Top 1 Accuracy | 84.5% | # 4 | ||
| Number of Params | 85M | # 33 | |||||
| Top 1 Accuracy (kNN, k=20) | 82.1% | # 4 | |||||
| Self-Supervised Image Classification | ImageNet | DINOv2 distilled (ViT-L/14) | Top 1 Accuracy | 86.3% | # 3 | ||
| Number of Params | 307M | # 13 | |||||
| Top 1 Accuracy (kNN, k=20) | 83.5% | # 2 | |||||
| Self-Supervised Image Classification | ImageNet | DINOv2 (ViT-g/14) | Top 1 Accuracy | 86.5% | # 2 | ||
| Number of Params | 1100M | # 2 | |||||
| Top 1 Accuracy (kNN, k=20) | 83.5% | # 2 | |||||
| Domain Generalization | ImageNet-C | DINOv2 (ViT-S/14, frozen model, linear eval) | mean Corruption Error (mCE) | 54.4 | # 24 | ||
| Number of params | 21M | # 24 | |||||
| Domain Generalization | ImageNet-C | DINOv2 (ViT-g/14, frozen model, linear eval) | mean Corruption Error (mCE) | 28.2 | # 1 | ||
| Number of params | 1100M | # 37 | |||||
| Domain Generalization | ImageNet-C | DINOv2 (ViT-L/14, frozen model, linear eval) | mean Corruption Error (mCE) | 31.5 | # 3 | ||
| Number of params | 307M | # 33 | |||||
| Domain Generalization | ImageNet-C | DINOv2 (ViT-B/14, frozen model, linear eval) | mean Corruption Error (mCE) | 42.7 | # 14 | ||
| Number of params | 85M | # 28 | |||||
| Self-Supervised Image Classification | ImageNet (finetuned) | DINOv2 (ViT-g/14) | Number of Params | 1100M | # 2 | ||
| Top 1 Accuracy | 88.5% | # 3 | |||||
| Self-Supervised Image Classification | ImageNet (finetuned) | DINOv2 (ViT-g/14, 448) | Number of Params | 1100M | # 2 | ||
| Top 1 Accuracy | 88.9% | # 1 | |||||
| Monocular Depth Estimation | KITTI Eigen split | DINOv2 (ViT-g/14 frozen, w/ DPT decoder) | absolute relative error | 0.0652 | # 22 | ||
| RMSE | 2.1128 | # 11 | |||||
| Sq Rel | 0.1797 | # 4 | |||||
| RMSE log | 0.0882 | # 17 | |||||
| Delta < 1.25 | 0.968 | # 13 | |||||
| Delta < 1.25^2 | 0.997 | # 5 | |||||
| Delta < 1.25^3 | 0.9993 | # 2 | |||||
| Monocular Depth Estimation | NYU-Depth V2 | DINOv2 (ViT-g/14 frozen, w/ DPT decoder) | RMSE | 0.279 | # 4 | ||
| absolute relative error | 0.0907 | # 14 | |||||
| Delta < 1.25 | 0.9497 | # 4 | |||||
| Delta < 1.25^2 | 0.996 | # 1 | |||||
| Delta < 1.25^3 | 0.9994 | # 1 | |||||
| log 10 | 0.0371 | # 10 | |||||
| Depth Estimation | NYU-Depth V2 | DINOv2 (ViT-g/14 frozen, w/ DPT decoder) | RMS | 0.279 | # 1 | ||
| Fine-Grained Image Classification | Oxford-IIIT Pet Dataset | DINOv2 (ViT-g/14, frozen model, linear eval) | Accuracy | 96.7 | # 1 |
