Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semantic Segmentation ADE20K XCiT-M24/8 (Semantic-FPN) Validation mIoU 46.9 # 95
Semantic Segmentation ADE20K XCiT-S24/8 (UperNet) Validation mIoU 48.1 # 80
Semantic Segmentation ADE20K XCiT-S24/8 (Semantic-FPN) Validation mIoU 47.1 # 93
Semantic Segmentation ADE20K XCiT-S12/8 (UperNet) Validation mIoU 46.6 # 100
Semantic Segmentation ADE20K XCiT-S12/8 (Semantic-FPN) Validation mIoU 44.2 # 128
Semantic Segmentation ADE20K XCiT-M24/8 (UperNet) Validation mIoU 48.4 # 75
Instance Segmentation COCO minival XCiT-S24/8 mask AP 43.0 # 36
Object Detection COCO minival XCiT-S24/8 box AP 48.1 # 59
Instance Segmentation COCO minival XCiT-M24/8 mask AP 43.7 # 34
Object Detection COCO minival XCiT-M24/8 box AP 48.5 # 56
Image Classification ImageNet XCiT-M24 Top 1 Accuracy 85.8% # 113
Number of params 84M # 590
GFLOPs 188 # 367
Image Classification ImageNet XCiT-S24 Top 1 Accuracy 85.6% # 124
Number of params 48M # 508
GFLOPs 106 # 353
Image Classification ImageNet XCiT-S12 Top 1 Accuracy 85.1% # 148
Number of params 26M # 430
GFLOPs 55.6 # 339
Image Classification ImageNet XCiT-L24 Top 1 Accuracy 86% # 106
Number of params 189M # 647
GFLOPs 417.9 # 383

Methods