SwinFIR: Revisiting the SwinIR with Fast Fourier Convolution and Improved Training for Image Super-Resolution

24 Aug 2022  ·  Dafeng Zhang, Feiyu Huang, Shizhuo Liu, Xiaobing Wang, Zhezhu Jin ·

Transformer-based methods have achieved impressive image restoration performance due to their capacities to model long-range dependency compared to CNN-based methods. However, advances like SwinIR adopts the window-based and local attention strategy to balance the performance and computational overhead, which restricts employing large receptive fields to capture global information and establish long dependencies in the early layers. To further improve the efficiency of capturing global information, in this work, we propose SwinFIR to extend SwinIR by replacing Fast Fourier Convolution (FFC) components, which have the image-wide receptive field. We also revisit other advanced techniques, i.e, data augmentation, pre-training, and feature ensemble to improve the effect of image reconstruction. And our feature ensemble method enables the performance of the model to be considerably enhanced without increasing the training and testing time. We applied our algorithm on multiple popular large-scale benchmarks and achieved state-of-the-art performance comparing to the existing methods. For example, our SwinFIR achieves the PSNR of 32.83 dB on Manga109 dataset, which is 0.8 dB higher than the state-of-the-art SwinIR method.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Super-Resolution BSD100 - 2x upscaling HAT_FIR PSNR 32.71 # 3
Image Super-Resolution BSD100 - 2x upscaling SwinFIR PSNR 32.64 # 5
SSIM 0.9054 # 4
Image Super-Resolution BSD100 - 3x upscaling HAT_FIR PSNR 29.6 # 2
Image Super-Resolution BSD100 - 3x upscaling SwinFIR PSNR 29.55 # 4
SSIM 0.8169 # 3
Image Super-Resolution BSD100 - 4x upscaling SwinFIR PSNR 28.03 # 5
SSIM 0.7520 # 8
Image Super-Resolution BSD100 - 4x upscaling HAT_FIR PSNR 28.07 # 3
Stereo Image Super-Resolution Flickr1024 - 2x upscaling SwinFIRSSR PSNR 30.14 # 1
Stereo Image Super-Resolution Flickr1024 - 4x upscaling SwinFIRSSR PSNR 24.29 # 1
Stereo Image Super-Resolution KITTI2012 - 2x upscaling SwinFIRSSR PSNR 31.79 # 1
Stereo Image Super-Resolution KITTI2012 - 4x upscaling SwinFIRSSR PSNR 27.16 # 1
Stereo Image Super-Resolution KITTI2015 - 2x upscaling SwinFIRSSR PSNR 31.45 # 1
Stereo Image Super-Resolution KITTI2015 - 4x upscaling SwinFIRSSR PSNR 26.89 # 3
Image Super-Resolution Manga109 - 2x upscaling SwinFIR PSNR 40.61 # 4
SSIM 0.9816 # 3
Image Super-Resolution Manga109 - 2x upscaling HAT_FIR PSNR 40.77 # 2
Image Super-Resolution Manga109 - 3x upscaling HAT_FIR PSNR 35.92 # 2
Image Super-Resolution Manga109 - 3x upscaling SwinFIR PSNR 35.77 # 4
SSIM 0.9563 # 3
Image Super-Resolution Manga109 - 4x upscaling SwinFIR PSNR 32.83 # 5
SSIM 0.9314 # 4
Image Super-Resolution Manga109 - 4x upscaling HAT_FIR PSNR 33.03 # 3
Stereo Image Super-Resolution Middlebury - 2x upscaling SwinFIRSSR PSNR 36.52 # 1
Stereo Image Super-Resolution Middlebury - 4x upscaling SwinFIRSSR PSNR 30.44 # 1
Image Super-Resolution Set14 - 2x upscaling SwinFIR PSNR 34.93 # 4
SSIM 0.9276 # 4
Image Super-Resolution Set14 - 2x upscaling HAT_FIR PSNR 35.17 # 2
Image Super-Resolution Set14 - 3x upscaling HAT_FIR PSNR 31.37 # 2
Image Super-Resolution Set14 - 3x upscaling SwinFIR PSNR 31.24 # 4
SSIM 0.8566 # 3
Image Super-Resolution Set14 - 4x upscaling SwinFIR PSNR 29.36 # 5
SSIM 0.7993 # 8
Image Super-Resolution Set14 - 4x upscaling HAT_FIR PSNR 29.44 # 3
Image Super-Resolution Set5 - 2x upscaling HAT_FIR PSNR 38.74 # 2
Image Super-Resolution Set5 - 2x upscaling SwinFIR PSNR 38.65 # 4
SSIM 0.9633 # 4
Image Super-Resolution Set5 - 3x upscaling SwinFIR PSNR 35.15 # 4
SSIM 0.9330 # 3
Image Super-Resolution Set5 - 3x upscaling HAT_FIR PSNR 35.21 # 2
Image Super-Resolution Urban100 - 2x upscaling SwinFIR PSNR 34.57 # 4
SSIM 0.9473 # 3
Image Super-Resolution Urban100 - 2x upscaling HAT_FIR PSNR 34.94 # 2
Image Super-Resolution Urban100 - 3x upscaling SwinFIR PSNR 30.43 # 4
SSIM 0.8913 # 3
Image Super-Resolution Urban100 - 3x upscaling HAT_FIR PSNR 30.77 # 2
Image Super-Resolution Urban100 - 4x upscaling SwinFIR PSNR 28.12 # 5
SSIM 0.8393 # 5
Image Super-Resolution Urban100 - 4x upscaling HAT_FIR PSNR 28.43 # 3

Methods