The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for "LLM-like" scaling in vision, and provides key steps towards getting there.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Image Classification ImageNet ViT-B/16 Top 1 Accuracy 88.6% # 44
Number of params 86M # 814
Image Classification ImageNet ViT-L/16 (384res, distilled from ViT-22B) Top 1 Accuracy 89.6% # 24
Number of params 307M # 915
Zero-Shot Transfer Image Classification ImageNet LiT-22B Accuracy (Private) 85.9 # 4
Zero-Shot Transfer Image Classification ImageNet-A LiT-22B Accuracy (Private) 90.1 # 2
Zero-Shot Transfer Image Classification ImageNet-R LiT-22B Accuracy 96.0 # 4
Zero-Shot Transfer Image Classification ImageNet V2 LiT-22B Accuracy (Private) 80.9 # 2
Action Classification Kinetics-400 ViT-22B Acc@1 88.0 # 22
Zero-Shot Transfer Image Classification ObjectNet LiT-22B Accuracy (Private) 87.6 # 1


No methods listed for this paper. Add relevant methods here