Uformer: A General U-Shaped Transformer for Image Restoration

In this paper, we present Uformer, an effective and efficient Transformer-based architecture for image restoration, in which we build a hierarchical encoder-decoder network using the Transformer block. In Uformer, there are two core designs. First, we introduce a novel locally-enhanced window (LeWin) Transformer block, which performs nonoverlapping window-based self-attention instead of global self-attention. It significantly reduces the computational complexity on high resolution feature map while capturing local context. Second, we propose a learnable multi-scale restoration modulator in the form of a multi-scale spatial bias to adjust features in multiple layers of the Uformer decoder. Our modulator demonstrates superior capability for restoring details for various image restoration tasks while introducing marginal extra parameters and computational cost. Powered by these two designs, Uformer enjoys a high capability for capturing both local and global dependencies for image restoration. To evaluate our approach, extensive experiments are conducted on several image restoration tasks, including image denoising, motion deblurring, defocus deblurring and deraining. Without bells and whistles, our Uformer achieves superior or comparable performance compared with the state-of-the-art algorithms. The code and models are available at https://github.com/ZhendongWang6/Uformer.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Denoising DND Uformer-B PSNR (sRGB) 39.98 # 2
SSIM (sRGB) 0.955 # 4
Image Defocus Deblurring DPD (Dual-view) Uformer-B PSNR 26.28 # 5
SSIM 0.891 # 1
Image Deblurring GoPro Uformer-B PSNR 32.97 # 8
SSIM 0.967 # 1
Params (M) 50.88 # 11
Deblurring GoPro Uformer-B PSNR 32.97 # 15
SSIM 0.967 # 3
Deblurring HIDE (trained on GOPRO) Uformer-B PSNR (sRGB) 30.83 # 8
SSIM (sRGB) 0.952 # 2
Params (M) 50.88 # 9
Deblurring RealBlur-J (trained on GoPro) Uformer-B PSNR (sRGB) 29.06 # 1
SSIM (sRGB) 0.884 # 1
Deblurring RealBlur-R (trained on GoPro) Uformer-B PSNR (sRGB) 36.22 # 1
SSIM (sRGB) 0.957 # 1
Deblurring RSBlur Uformer-B Average PSNR 33.98 # 1
Image Denoising SIDD Uformer-B PSNR (sRGB) 39.89 # 6
SSIM (sRGB) 0.960 # 3
Image Dehazing SOTS Indoor Uformer PSNR 31.91 # 11
SSIM 0.971 # 11
Image Dehazing SOTS Outdoor Uformer PSNR 26.52 # 9
SSIM 0.945 # 8
Image Enhancement TIP 2018 Uformer-B PSNR 29.28 # 4
SSIM 0.917 # 2

Methods