Emerging Properties in Self-Supervised Vision Transformers

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Place Recognition 17 Places DINO Recall@1 61.82 # 3
Copy Detection Copydays strong subset DINO (ViT-B/8) mAP 85.5 # 2
Video Object Segmentation DAVIS 2017 DINO (ViT-B/8, ImageNet retrain) J&F 71.4 # 2
Visual Place Recognition Gardens Point DINO Recall@1 78.50 # 3
Self-Supervised Image Classification ImageNet DINO (ViT-S/16) Top 1 Accuracy 77.0% # 51
Number of Params 21M # 77
Self-Supervised Image Classification ImageNet DINO (ViT-B/16) Top 1 Accuracy 78.2% # 43
Number of Params 85M # 38
Self-Supervised Image Classification ImageNet DINO (ResNet-50) Top 1 Accuracy 75.3% # 69
Number of Params 24M # 48
Self-Supervised Image Classification ImageNet DINO (xcit_medium_24_p8) Top 1 Accuracy 80.3% # 23
Number of Params 84M # 42
Self-Supervised Image Classification ImageNet DINO (ViT-B/8) Top 1 Accuracy 80.1% # 26
Self-Supervised Image Classification ImageNet DINO (ViT-S/8) Top 1 Accuracy 79.7% # 30
Number of Params 21M # 77
Self-Supervised Image Classification ImageNet (finetuned) DINO (ViT-B/16) Number of Params 85M # 39
Top 1 Accuracy 82.8% # 48
Image Classification OmniBenchmark DINO Average Top-1 Accuracy 38.9 # 8

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Visual Place Recognition Baidu Mall DINO Recall@1 48.30 # 6
Visual Place Recognition Hawkins DINO Recall@1 46.61 # 2
Visual Place Recognition Laurel Caverns DINO Recall@1 41.07 # 2
Visual Place Recognition Mid-Atlantic Ridge DINO Recall@1 27.72 # 2
Visual Place Recognition Nardo-Air DINO Recall@1 57.75 # 3
Visual Place Recognition Nardo-Air R DINO Recall@1 84.51 # 4
Visual Place Recognition Oxford RobotCar Dataset DINO Recall@1 15.71 # 7
Visual Place Recognition Pittsburgh-30k-test DINO Recall@1 70.13 # 11
Visual Place Recognition St Lucia DINO Recall@1 45.22 # 8
Visual Place Recognition VP-Air DINO Recall@1 24.02 # 4

Methods