Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model

1 Dec 2022  ·  Yinhuai Wang, Jiwen Yu, Jian Zhang ·

Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we can yield diverse results satisfying both data consistency and realness. We further propose an enhanced and robust version, dubbed DDNM+, to support noisy restoration and improve restoration quality for hard tasks. Our experiments on several IR tasks reveal that DDNM outperforms other state-of-the-art zero-shot IR methods. We also demonstrate that DDNM+ can solve complex real-world applications, e.g., old photo restoration.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Colorization CelebA DDRM Consistency 455.9 # 1
FID 31.26 # 2
Image Deblurring CelebA DDRM PSNR 43.07 # 2
SSIM 0.993 # 2
FID 6.24 # 2
Image Super-Resolution CelebA DDNM PSNR 31.63 # 1
SSIM 0.945 # 1
FID 22.27 # 1
Image Super-Resolution CelebA DDRM PSNR 31.63 # 1
SSIM 0.945 # 1
FID 31.04 # 3
Image Super-Resolution CelebA ILVR PSNR 31.59 # 3
SSIM 0.945 # 1
FID 29.82 # 2
Image Super-Resolution CelebA PULSE PSNR 22.74 # 5
SSIM 0.623 # 5
FID 40.33 # 4
Image Inpainting CelebA DDNM PSNR 35.64 # 1
SSIM 0.982 # 1
FID 4.54 # 1
Image Inpainting CelebA RePaint PSNR 35.2 # 2
SSIM 0.981 # 2
FID 14.19 # 3
Image Inpainting CelebA A+y PSNR 15.57 # 4
SSIM 0.809 # 4
FID 181.56 # 4
Image Compressed Sensing CelebA A+y PSNR 15.09 # 3
SSIM 0.583 # 3
FID 377.7 # 3
Image Colorization CelebA A+y Consistency 0 # 3
FID 68.81 # 3
Image Deblurring CelebA A+y PSNR 18.85 # 3
SSIM 0.741 # 3
FID 54.31 # 3
Image Super-Resolution CelebA A+y PSNR 27.27 # 4
SSIM 0.782 # 4
FID 103.3 # 5
Image Inpainting CelebA DDRM PSNR 34.79 # 3
SSIM 0.978 # 3
FID 12.53 # 2
Image Compressed Sensing CelebA DDNM PSNR 27.56 # 1
SSIM 0.909 # 1
FID 28.8 # 1
Image Compressed Sensing CelebA DDRM PSNR 24.86 # 2
SSIM 0.876 # 2
FID 46.77 # 2
Image Colorization CelebA DDNM Consistency 26.25 # 2
FID 26.44 # 1
Image Deblurring CelebA DDNM PSNR 46.72 # 1
SSIM 0.996 # 1
FID 1.41 # 1
Image Deblurring ImageNet A+y PSNR 18.56 # 3
SSIM 0.6616 # 3
FID 55.42 # 3
Image Super-Resolution ImageNet A+y PSNR 24.26 # 4
SSIM 0.684 # 5
FID 134.4 # 5
Image Inpainting ImageNet DDNM PSNR 32.06 # 1
SSIM 0.968 # 1
FID 3.89 # 1
Image Inpainting ImageNet DDRM PSNR 31.73 # 3
SSIM 0.966 # 2
FID 4.82 # 2
Image Inpainting ImageNet RePaint PSNR 31.87 # 2
SSIM 0.963 # 3
FID 12.31 # 3
Image Compressed Sensing ImageNet DDNM PSNR 21.66 # 1
SSIM 0.749 # 1
FID 64.68 # 1
Image Compressed Sensing ImageNet DDRM PSNR 19.95 # 2
SSIM 0.704 # 2
FID 97.99 # 2
Image Colorization ImageNet DDNM Consistency 42.32 # 2
FID 36.32 # 1
Image Colorization ImageNet DDRM Consistency 260.4 # 1
FID 36.56 # 2
Image Colorization ImageNet DGP FID 69.54 # 4
Image Deblurring ImageNet DDNM PSNR 44.93 # 1
SSIM 0.994 # 1
FID 1.15 # 1
Image Deblurring ImageNet DDRM PSNR 43.01 # 2
SSIM 0.992 # 2
FID 1.48 # 2
Image Super-Resolution ImageNet DDNM PSNR 27.46 # 1
SSIM 0.87 # 1
FID 39.26 # 1
Image Super-Resolution ImageNet DDRM PSNR 27.38 # 3
SSIM 0.869 # 3
FID 43.15 # 2
Image Super-Resolution ImageNet ILVR PSNR 27.4 # 2
SSIM 0.87 # 1
FID 43.66 # 3
Image Super-Resolution ImageNet DGP PSNR 23.18 # 5
SSIM 0.798 # 4
FID 64.34 # 4
Image Inpainting ImageNet A+y PSNR 14.52 # 4
SSIM 0.799 # 4
FID 72.71 # 4
Image Compressed Sensing ImageNet A+y PSNR 15.65 # 3
SSIM 0.51 # 3
FID 277.4 # 3
Image Colorization ImageNet A+y Consistency 0 # 3
FID 43.37 # 3

Methods