Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation

Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to convolutional neural network (CNN)-based models. However, ViTs are mainly designed for image classification that generate single-scale low-resolution representations, which makes dense prediction tasks such as semantic segmentation challenging for ViTs. Therefore, we propose HRViT, which enhances ViTs to learn semantically-rich and spatially-precise multi-scale representations by integrating high-resolution multi-branch architectures with ViTs. We balance the model performance and efficiency of HRViT by various branch-block co-optimization techniques. Specifically, we explore heterogeneous branch designs, reduce the redundancy in linear layers, and augment the attention block with enhanced expressiveness. Those approaches enabled HRViT to push the Pareto frontier of performance and efficiency on semantic segmentation to a new level, as our evaluation results on ADE20K and Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction, demonstrating the potential of HRViT as a strong vision backbone for semantic segmentation.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semantic Segmentation ADE20K HRViT-b1 (SegFormer, SS) Validation mIoU 45.88 # 160
Params (M) 8.2 # 57
GFLOPs (512 x 512) 14.6 # 1
Semantic Segmentation ADE20K HRViT-b2 (SegFormer, SS) Validation mIoU 48.76 # 118
Params (M) 20.8 # 53
GFLOPs (512 x 512) 28.0 # 3
Semantic Segmentation ADE20K HRViT-b3 (SegFormer, SS) Validation mIoU 50.2 # 93
Params (M) 28.7 # 50
GFLOPs (512 x 512) 67.9 # 6
Semantic Segmentation Cityscapes val HRViT-b3 (SegFormer, SS) mIoU 83.16% # 20
Semantic Segmentation Cityscapes val HRViT-b1 (SegFormer, SS) mIoU 81.63% # 30
Semantic Segmentation Cityscapes val HRViT-b2 (SegFormer, SS) mIoU 82.81% # 22


No methods listed for this paper. Add relevant methods here