UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition

27 Nov 2023  ·  Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao, Lin Song, Xiangyu Yue, Ying Shan ·

Large-kernel convolutional neural networks (ConvNets) have recently received extensive research attention, but there are two unresolved and critical issues that demand further investigation. 1) The architectures of existing large-kernel ConvNets largely follow the design principles of conventional ConvNets or transformers, while the architectural design for large-kernel ConvNets remains under-addressed. 2) As transformers have dominated multiple modalities, it remains to be investigated whether ConvNets also have a strong universal perception ability in domains beyond vision. In this paper, we contribute from two aspects. 1) We propose four architectural guidelines for designing large-kernel ConvNets, the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep. Following such guidelines, our proposed large-kernel ConvNet shows leading performance in image recognition. For example, our models achieve an ImageNet accuracy of 88.0%, ADE20K mIoU of 55.6%, and COCO box AP of 56.4%, demonstrating better performance and higher speed than a number of recently proposed powerful competitors. 2) We discover that large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient. With certain modality-related preprocessing approaches, the proposed model achieves state-of-the-art performance on time-series forecasting and audio recognition tasks even without modality-specific customization to the architecture. Code and all the models at https://github.com/AILab-CVC/UniRepLKNet.

PDF Abstract

Results from the Paper

 Ranked #1 on Object Detection on COCO 2017 (mAP metric)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semantic Segmentation ADE20K UniRepLKNet-T Validation mIoU 49.1 # 130
Semantic Segmentation ADE20K UniRepLKNet-XL Validation mIoU 55.6 # 40
Semantic Segmentation ADE20K UniRepLKNet-L++ Validation mIoU 55 # 45
Semantic Segmentation ADE20K UniRepLKNet-B++ Validation mIoU 53.9 # 66
Semantic Segmentation ADE20K UniRepLKNet-S++ Validation mIoU 52.7 # 82
Semantic Segmentation ADE20K UniRepLKNet-S Validation mIoU 51 # 96
Object Detection COCO 2017 UniRepLKNet-XL++ mAP 56.4 # 1
Object Detection COCO 2017 UniRepLKNet-L++ mAP 55.8 # 2
Object Detection COCO 2017 UniRepLKNet-B++ mAP 54.8 # 3
Object Detection COCO 2017 UniRepLKNet-S++ mAP 54.3 # 4
Object Detection COCO 2017 UniRepLKNet-S mAP 53 # 6
Object Detection COCO 2017 UniRepLKNet-T mAP 51.7 # 8
Image Classification ImageNet UniRepLKNet-XL++ Top 1 Accuracy 88% # 69
Image Classification ImageNet UniRepLKNet-L++ Top 1 Accuracy 87.9% # 74
Image Classification ImageNet UniRepLKNet-B++ Top 1 Accuracy 87.4% # 93
Image Classification ImageNet UniRepLKNet-S++ Top 1 Accuracy 86.4% # 143
Image Classification ImageNet UniRepLKNet-S Top 1 Accuracy 83.9% # 342
Image Classification ImageNet UniRepLKNet-T Top 1 Accuracy 83.2% # 407
Image Classification ImageNet UniRepLKNet-N Top 1 Accuracy 81.6% # 561
Image Classification ImageNet UniRepLKNet-P Top 1 Accuracy 80.2% # 647
Image Classification ImageNet UniRepLKNet-F Top 1 Accuracy 78.6% # 745
Image Classification ImageNet UniRepLKNet-A Top 1 Accuracy 77% # 814