Revisiting Temporal Modeling for Video Super-resolution

13 Aug 2020  ·  Takashi Isobe, Fang Zhu, Xu Jia, Shengjin Wang ·

Video super-resolution plays an important role in surveillance video analysis and ultra-high-definition video display, which has drawn much attention in both the research and industrial communities. Although many deep learning-based VSR methods have been proposed, it is hard to directly compare these methods since the different loss functions and training datasets have a significant impact on the super-resolution results... In this work, we carefully study and compare three temporal modeling methods (2D CNN with early fusion, 3D CNN with slow fusion and Recurrent Neural Network) for video super-resolution. We also propose a novel Recurrent Residual Network (RRN) for efficient video super-resolution, where residual learning is utilized to stabilize the training of RNN and meanwhile to boost the super-resolution performance. Extensive experiments show that the proposed RRN is highly computational efficiency and produces temporal consistent VSR results with finer details than other temporal modeling methods. Besides, the proposed method achieves state-of-the-art results on several widely used benchmarks. read more

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Super-Resolution MSU Video Super Resolution Benchmark RRN-10L Subjective score 3.592 # 12
ERQAv1.0 0.627 # 19
QRCRv1.0 0.557 # 6
SSIM 0.790 # 19
CRRMv1.0 0.989 # 11
PSNR 24.252 # 21
FPS 0.390 # 2
Video Super-Resolution MSU Video Super Resolution Benchmark RRN-5L Subjective score 3.253 # 17
ERQAv1.0 0.617 # 20
QRCRv1.0 0.549 # 11
SSIM 0.789 # 20
CRRMv1.0 0.989 # 11
PSNR 23.786 # 22
FPS 0.365 # 1
Video Super-Resolution SPMCS - 4x upscaling RRN-L PSNR 29.84 # 1
SSIM 0.8690 # 1
Video Super-Resolution UDM10 - 4x upscaling RRN-L PSNR 38.97 # 1
SSIM 0.9534 # 1
Video Super-Resolution Vid4 - 4x upscaling RRN-L PSNR 27.69 # 1
SSIM 0.8209 # 5

Methods


No methods listed for this paper. Add relevant methods here