A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift

CVPR 2022  ยท  Shi Guo, Xi Yang, jianqi ma, Gaofeng Ren, Lei Zhang ยท

Denoising and demosaicking are two essential steps to reconstruct a clean full-color image from the raw data. Recently, joint denoising and demosaicking (JDD) for burst images, namely JDD-B, has attracted much attention by using multiple raw images captured in a short time to reconstruct a single high-quality image. One key challenge of JDD-B lies in the robust alignment of image frames. State-of-the-art alignment methods in feature domain cannot effectively utilize the temporal information of burst images, where large shifts commonly exist due to camera and object motion. In addition, the higher resolution (e.g., 4K) of modern imaging devices results in larger displacement between frames. To address these challenges, we design a differentiable two-stage alignment scheme sequentially in patch and pixel level for effective JDD-B. The input burst images are firstly aligned in the patch level by using a differentiable progressive block matching method, which can estimate the offset between distant frames with small computational cost. Then we perform implicit pixel-wise alignment in full-resolution feature domain to refine the alignment results. The two stages are jointly trained in an end-to-end manner. Extensive experiments demonstrate the significant improvement of our method over existing JDD-B methods. Codes are available at https://github.com/GuoShi28/2StageAlign.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Datasets


Introduced in the Paper:

Videezy4K

Used in the Paper:

REDS SC_burst
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Joint Demosaicing and Denoising 5 frames with input 2160x3840 EDVR Runtime(s) 10.4 # 1
Joint Demosaicing and Denoising 5 frames with input 2160x3840 Two-Stage Runtime(s) 19.7 # 3
Joint Demosaicing and Denoising 5 frames with input 2160x3840 GCP-Net Runtime(s) 19.5 # 2
Joint Demosaicing and Denoising 5 frames with input 2160x3840 RviDeNet Runtime(s) 124.3 # 4
Joint Demosaicing and Denoising REDS EDVR PSNR/SSIM 34.02 # 4
Joint Demosaicing and Denoising REDS Two-Stage PSNR/SSIM 36.59 # 1
Joint Demosaicing and Denoising REDS GCP-Net PSNR/SSIM 36.2 # 2
Joint Demosaicing and Denoising REDS RviDeNet PSNR/SSIM 34.86 # 3
Joint Demosaicing and Denoising Videezy4K Two-Stage PSNR 38.74 # 1
Joint Demosaicing and Denoising Videezy4K GCP-Net PSNR 37.94 # 2
Joint Demosaicing and Denoising Videezy4K RviDeNet PSNR 37.5 # 3
Joint Demosaicing and Denoising Videezy4K EDVR PSNR 37.19 # 4

Methods


No methods listed for this paper. Add relevant methods here