Bottleneck Transformers for Visual Recognition

We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency... (read more)

PDF Abstract

Datasets


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Object Detection COCO minival BoTNet 200 (Mask R-CNN, single scale, 72 epochs) box AP 49.7 # 12
AP50 71.3 # 2
AP75 54.6 # 8
Instance Segmentation COCO minival BoTNet 200 (Mask R-CNN, single scale, 72 epochs) mask AP 44.4 # 10
Instance Segmentation COCO minival BoTNet 152 (Mask R-CNN, single scale, 72 epochs) mask AP 43.7 # 12
Object Detection COCO minival BoTNet 50 (72 epochs) box AP 45.9 # 26
Instance Segmentation COCO minival BoTNet 50 (72 epochs) mask AP 40.7 # 19
Object Detection COCO minival BoTNet 152 (Mask R-CNN, single scale, 72 epochs) box AP 49.5 # 13
AP50 71 # 3
AP75 54.2 # 9
Image Classification ImageNet SENet-350 Top 1 Accuracy 83.8% # 87
Top 5 Accuracy 96.6% # 48
Image Classification ImageNet BoTNet T6 Top 1 Accuracy 84% # 76
Top 5 Accuracy 96.7% # 44
Number of params 53.9M # 86
Image Classification ImageNet BoTNet T5 Top 1 Accuracy 83.5% # 93
Top 5 Accuracy 96.5% # 51
Number of params 75.1M # 59
Image Classification ImageNet BoTNet T4 Top 1 Accuracy 82.8% # 116
Top 5 Accuracy 96.3% # 58
Number of params 54.7M # 85
Image Classification ImageNet ResNet-50 Top 1 Accuracy 78.8% # 228
Top 5 Accuracy 94.5% # 114
Number of params 25.5M # 129
Image Classification ImageNet SENet-152 Top 1 Accuracy 82.2% # 139
Top 5 Accuracy 95.9% # 67
Number of params 66.6M # 67
Image Classification ImageNet SENet-50 Top 1 Accuracy 79.4% # 205
Top 5 Accuracy 94.6% # 108
Number of params 28.02M # 119
Image Classification ImageNet BoTNet T7-320 Top 1 Accuracy 84.2% # 68
Top 5 Accuracy 96.9% # 37
Number of params 75.1M # 59
Image Classification ImageNet ResNet-101 Top 1 Accuracy 80% # 191
Top 5 Accuracy 95% # 94
Number of params 44.4M # 91
Image Classification ImageNet BoTNet T7 Top 1 Accuracy 84.7% # 59
Top 5 Accuracy 97% # 33
Number of params 75.1M # 59
Image Classification ImageNet BoTNet T3 Top 1 Accuracy 81.7% # 153
Top 5 Accuracy 95.8% # 69
Number of params 33.5M # 112
Image Classification ImageNet SENet-101 Top 1 Accuracy 81.4% # 161
Top 5 Accuracy 95.7% # 74
Number of params 49.2M # 87

Methods used in the Paper


METHOD TYPE
Channel-wise Soft Attention
Attention Mechanisms
Batch Normalization
Normalization
Split Attention
Image Model Blocks
Max Pooling
Pooling Operations
1x1 Convolution
Convolutions
Pointwise Convolution
Convolutions
ResNeSt
Image Models
Scaled Dot-Product Attention
Attention Mechanisms
Residual Connection
Skip Connections
Average Pooling
Pooling Operations
Convolution
Convolutions
ReLU
Activation Functions
Dense Connections
Feedforward Networks
Squeeze-and-Excitation Block
Image Model Blocks
SiLU
Activation Functions
Label Smoothing
Regularization
Weight Decay
Regularization
RandAugment
Image Data Augmentation
Random Resized Crop
Image Data Augmentation
Cosine Annealing
Learning Rate Schedules
SGD with Momentum
Stochastic Optimization
Bottleneck Transformer Block
Image Model Blocks
Bottleneck Transformer
Image Models
Sigmoid Activation
Activation Functions
Softmax
Output Functions
RoIAlign
RoI Feature Extractors
Mask R-CNN
Instance Segmentation Models