An Implicit Alignment for Video Super-Resolution

29 Apr 2023  ·  Kai Xu, Ziwei Yu, Xin Wang, Michael Bi Mi, Angela Yao ·

Video super-resolution commonly uses a frame-wise alignment to support the propagation of information over time. The role of alignment is well-studied for low-level enhancement in video, but existing works have overlooked one critical step -- re-sampling. Most works, regardless of how they compensate for motion between frames, be it flow-based warping or deformable convolution/attention, use the default choice of bilinear interpolation for re-sampling. However, bilinear interpolation acts effectively as a low-pass filter and thus hinders the aim of recovering high-frequency content for super-resolution. This paper studies the impact of re-sampling on alignment for video super-resolution. Extensive experiments reveal that for alignment to be effective, the re-sampling should preserve the original sharpness of the features and prevent distortions. From these observations, we propose an implicit alignment method that re-samples through a window-based cross-attention with sampling positions encoded by sinusoidal positional encoding. The re-sampling is implicitly computed by learned network weights. Experiments show that the proposed implicit alignment enhances the performance of state-of-the-art frameworks with minimal impact on both synthetic and real-world datasets.

PDF Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Super-Resolution REDS4- 4x upscaling IART PSNR 32.9 # 1
SSIM 0.9138 # 1


No methods listed for this paper. Add relevant methods here