S$^2$-FPN: Scale-ware Strip Attention Guided Feature Pyramid Network for Real-time Semantic Segmentation

15 Jun 2022  ·  Mohammed A. M. Elhassan, Chenhui Yang, Chenxi Huang, Tewodros Legesse Munea, Xin Hong ·

Modern high-performance semantic segmentation methods employ a heavy backbone and dilated convolution to extract the relevant feature. Although extracting features with both contextual and semantic information is critical for the segmentation tasks, it brings a memory footprint and high computation cost for real-time applications. This paper presents a new model to achieve a trade-off between accuracy/speed for real-time road scene semantic segmentation. Specifically, we proposed a lightweight model named Scale-aware Strip Attention Guided Feature Pyramid Network (S$^2$-FPN). Our network consists of three main modules: Attention Pyramid Fusion (APF) module, Scale-aware Strip Attention Module (SSAM), and Global Feature Upsample (GFU) module. APF adopts an attention mechanisms to learn discriminative multi-scale features and help close the semantic gap between different levels. APF uses the scale-aware attention to encode global context with vertical stripping operation and models the long-range dependencies, which helps relate pixels with similar semantic label. In addition, APF employs channel-wise reweighting block (CRB) to emphasize the channel features. Finally, the decoder of S$^2$-FPN then adopts GFU, which is used to fuse features from APF and the encoder. Extensive experiments have been conducted on two challenging semantic segmentation benchmarks, which demonstrate that our approach achieves better accuracy/speed trade-off with different model settings. The proposed models have achieved a results of 76.2\%mIoU/87.3FPS, 77.4\%mIoU/67FPS, and 77.8\%mIoU/30.5FPS on Cityscapes dataset, and 69.6\%mIoU,71.0\% mIoU, and 74.2\% mIoU on Camvid dataset. The code for this work will be made available at \url{https://github.com/mohamedac29/S2-FPN

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Real-Time Semantic Segmentation CamVid S^2-FPN34M mIoU 74.2 # 13
Real-Time Semantic Segmentation CamVid S^2-FPN34 mIoU 71.0 # 18
Frame (fps) 107.2 # 6
Real-Time Semantic Segmentation CamVid S^2-FPN18 mIoU 69.5 # 19
Frame (fps) 124.2 # 5
Real-Time Semantic Segmentation Cityscapes S^2-FPN34 mIoU 77.4 # 1
Semantic Segmentation Cityscapes SPFNet34M mIoU 77.8 # 1
Real-Time Semantic Segmentation Cityscapes S^2-FPN18 mIoU 76.2 # 2

Methods