Unicom: Universal and Compact Representation Learning for Image Retrieval

12 Apr 2023  ยท  Xiang An, Jiankang Deng, Kaicheng Yang, Jaiwei Li, Ziyong Feng, Jia Guo, Jing Yang, Tongliang Liu ยท

Modern image retrieval methods typically rely on fine-tuning pre-trained encoders to extract image-level descriptors. However, the most widely used models are pre-trained on ImageNet-1K with limited classes. The pre-trained feature representation is therefore not universal enough to generalize well to the diverse open-world classes. In this paper, we first cluster the large-scale LAION400M into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model. Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict. To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss. To further enhance the low-dimensional feature representation, we randomly select partial feature dimensions when calculating the similarities between embeddings and class-wise prototypes. The dual random partial selections are with respect to the class dimension and the feature dimension of the prototype matrix, making the classification conflict-robust and the feature embedding compact. Our method significantly outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks. The code and pre-trained models are released to facilitate future research https://github.com/deepglint/unicom.

PDF Abstract

Results from the Paper


 Ranked #1 on Image Retrieval on SOP (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Metric Learning CARS196 Unicom+ViT-L@336px R@1 98.2 # 1
Metric Learning CUB-200-2011 Unicom+ViT-L@336px R@1 90.1 # 1
Self-Supervised Image Classification ImageNet Unicom (ViT-L/14@336px) Top 1 Accuracy 82.7% # 10
Self-Supervised Image Classification ImageNet Unicom (ViT-L/14) Top 1 Accuracy 81.8% # 14
Self-Supervised Image Classification ImageNet Unicom (ViT-B/16) Top 1 Accuracy 79.1% # 36
Self-Supervised Image Classification ImageNet Unicom (ViT-B/32) Top 1 Accuracy 75.0% # 75
Image Classification ImageNet Unicom (ViT-L/14@336px) (Finetuned) Top 1 Accuracy 88.3 # 62
Image Retrieval iNaturalist Unicom+ViT-L@336px R@1 88.9 # 1
Metric Learning In-Shop Unicom+ViT-L@336px R@1 96.7 # 1
Image Retrieval SOP Unicom+ViT-L@336px R@1 91.2 # 1
Metric Learning Stanford Online Products Unicom+ViT-L@336px R@1 91.2 # 1

Methods


CLIP โ€ข Softmax