UniNet: Unified Architecture Search with Convolution, Transformer, and MLP

8 Oct 2021  ·  Jihao Liu, Hongsheng Li, Guanglu Song, Xin Huang, Yu Liu ·

Recently, transformer and multi-layer perceptron (MLP) architectures have achieved impressive results on various vision tasks. A few works investigated manually combining those operators to design visual network architectures, and can achieve satisfactory performances to some extent. In this paper, we propose to jointly search the optimal combination of convolution, transformer, and MLP for building a series of all-operator network architectures with high performances on visual tasks. We empirically identify that the widely-used strided convolution or pooling based down-sampling modules become the performance bottlenecks when the operators are combined to form a network. To better tackle the global context captured by the transformer and MLP operators, we propose two novel context-aware down-sampling modules, which can better adapt to the global information encoded by transformer and MLP operators. To this end, we jointly search all operators and down-sampling modules in a unified search space. Notably, Our searched network UniNet (Unified Network) outperforms state-of-the-art pure convolution-based architecture, EfficientNet, and pure transformer-based architecture, Swin-Transformer, on multiple public visual benchmarks, ImageNet classification, COCO object detection, and ADE20K semantic segmentation.

PDF Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Classification ImageNet UniNet-B4 Top 1 Accuracy 84.2% # 338
Number of params 73.5M # 824
GFLOPs 9.9 # 318
Image Classification ImageNet UniNet-B2 Top 1 Accuracy 82.7% # 504
Number of params 22.5M # 581
GFLOPs 2.4 # 163
Image Classification ImageNet UniNet-B5 Top 1 Accuracy 85.2% # 255
Number of params 73.5M # 824
GFLOPs 23.2 # 409
Image Classification ImageNet UniNet-B1 Top 1 Accuracy 80.4% # 694
Number of params 14M # 521
GFLOPs 0.99 # 104
Image Classification ImageNet UniNet-B0 Top 1 Accuracy 79.1% # 764
Number of params 11.9M # 503
GFLOPs 0.56 # 58

Methods