Recurrent Video Restoration Transformer with Guided Deformable Attention

Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability. In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT. RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency. Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature. Within each clip, different frame features are jointly updated with implicit feature aggregation. Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism. Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Denoising DAVIS sigma10 RVRT PSNR 40.57 # 3
Video Denoising DAVIS sigma20 RVRT PSNR 38.05 # 3
Video Denoising DAVIS sigma30 RVRT PSNR 36.57 # 2
Video Denoising DAVIS sigma40 RVRT PSNR 35.47 # 2
Video Denoising DAVIS sigma50 RVRT PSNR 34.57 # 2
Deblurring DVD RVRT PSNR 34.92 # 1
SSIM 97.38 # 1
Video Denoising Set8 sigma10 RVRT PSNR 37.53 # 3
Video Denoising Set8 sigma20 RVRT PSNR 34.83 # 3
Video Denoising Set8 sigma30 RVRT PSNR 33.3 # 3
Video Denoising Set8 sigma40 RVRT PSNR 32.21 # 2
Video Denoising Set8 sigma50 RVRT PSNR 31.33 # 2
Analog Video Restoration TAPE RVRT LPIPS 0.117 # 6
VMAF 72.41 # 4
PSNR 32.47 # 4
SSIM 0.896 # 6
Video Super-Resolution UDM10 - 4x upscaling RVRT PSNR 40.9 # 2
SSIM 0.9729 # 2
Video Super-Resolution Vid4 - 4x upscaling RVRT PSNR 27.99 # 3
SSIM 0.8462 # 3
Video Super-Resolution Vid4 - 4x upscaling - BD degradation RVRT PSNR 29.54 # 1
SSIM 0.8810 # 1
Video Super-Resolution Vimeo90K RVRT PSNR 38.59 # 2
SSIM 0.9576 # 1