VRT: A Video Restoration Transformer

28 Jan 2022  ·  Jingyun Liang, JieZhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, Luc van Gool ·

Video restoration (e.g., video super-resolution) aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which either is restricted by frame-by-frame restoration or lacks long-range modelling ability. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction and long-range temporal dependency modelling abilities. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal mutual self attention (TMSA) and parallel warping. TMSA divides the video into small clips, on which mutual attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on five tasks, including video super-resolution, video deblurring, video denoising, video frame interpolation and space-time video super-resolution, demonstrate that VRT outperforms the state-of-the-art methods by large margins ($\textbf{up to 2.16dB}$) on fourteen benchmark datasets.

PDF Abstract

Results from the Paper
Edit

 Ranked #1 on Deblurring on BASED

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data 		Result Benchmark
Deblurring BASED VRT (REDS) VMAF 66.81782 # 2
Compare
LPIPS 0.08248 # 2
Compare
ERQAv2.0 0.74874 # 1
Compare
SSIM 0.94503 # 1
Compare
PSNR 30.97878 # 3
Compare
Deblurring BASED VRT (GoPro) VMAF 66.72253 # 3
Compare
PSNR 31.42945 # 1
Compare
Video Denoising DAVIS sigma10 VRT PSNR 40.82 # 2
Compare
Video Denoising DAVIS sigma20 VRT PSNR 38.15 # 2
Compare
Video Denoising DAVIS sigma30 VRT PSNR 36.52 # 3
Compare
Video Denoising DAVIS sigma40 VRT PSNR 35.32 # 3
Compare
Video Denoising DAVIS sigma50 VRT PSNR 34.36 # 3
Compare
Deblurring DVD VRT PSNR 34.27 # 4
Compare
Deblurring GoPro VRT PSNR 34.81 # 6
Compare
SSIM 0.9724 # 5
Compare
Deblurring MSU BASED VRT (REDS) SSIM 0.94601 # 4
Compare
PSNR 30.97878 # 8
Compare
VMAF 66.81782 # 5
Compare
LPIPS 0.08248 # 5
Compare
ERQAv2.0 0.75056 # 1
Compare
Subjective 1.5660 # 3
Compare
Deblurring MSU BASED VRT (GoPro) SSIM 0.94503 # 6
Compare
PSNR 31.42945 # 4
Compare
VMAF 66.72253 # 6
Compare
LPIPS 0.08165 # 3
Compare
ERQAv2.0 0.74874 # 2
Compare
Subjective 2.3854 # 2
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression VRT + vvenc BSQ-rate over ERQA 18.333 # 74
Compare
BSQ-rate over Subjective Score 2.235 # 29
Compare
BSQ-rate over VMAF 0.652 # 2
Compare
BSQ-rate over PSNR 5.777 # 25
Compare
BSQ-rate over MS-SSIM 0.836 # 13
Compare
BSQ-rate over LPIPS 11.496 # 63
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression VRT + x265 BSQ-rate over ERQA 8.92 # 40
Compare
BSQ-rate over Subjective Score 2.023 # 28
Compare
BSQ-rate over VMAF 1.217 # 21
Compare
BSQ-rate over PSNR 6.634 # 37
Compare
BSQ-rate over MS-SSIM 1.257 # 24
Compare
BSQ-rate over LPIPS 11.329 # 60
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression VRT + aomenc BSQ-rate over ERQA 12.289 # 50
Compare
BSQ-rate over Subjective Score 2.631 # 34
Compare
BSQ-rate over VMAF 1.733 # 39
Compare
BSQ-rate over PSNR 10.075 # 53
Compare
BSQ-rate over MS-SSIM 2.797 # 43
Compare
BSQ-rate over LPIPS 4.429 # 37
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression VRT + uavs3e BSQ-rate over ERQA 6.619 # 30
Compare
BSQ-rate over Subjective Score 2.511 # 33
Compare
BSQ-rate over VMAF 1.425 # 29
Compare
BSQ-rate over PSNR 5.862 # 29
Compare
BSQ-rate over MS-SSIM 1.982 # 37
Compare
BSQ-rate over LPIPS 4.003 # 31
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression VRT + x264 BSQ-rate over ERQA 1.578 # 8
Compare
BSQ-rate over Subjective Score 1.245 # 17
Compare
BSQ-rate over VMAF 0.7 # 7
Compare
BSQ-rate over PSNR 1.09 # 7
Compare
BSQ-rate over MS-SSIM 0.662 # 3
Compare
BSQ-rate over LPIPS 1.259 # 12
Compare
Video Super-Resolution MSU Video Super Resolution Benchmark: Detail Restoration VRT Subjective score 7.628 # 1
Compare
ERQAv1.0 0.758 # 1
Compare
QRCRv1.0 0.722 # 1
Compare
SSIM 0.902 # 1
Compare
PSNR 31.669 # 1
Compare
FPS 2.778 # 4
Compare
1 - LPIPS 0.929 # 4
Compare
Video Super-Resolution MSU Video Upscalers: Quality Enhancement VRT-Reds-L PSNR 31.01 # 5
Compare
LPIPS 0.343 # 29
Compare
SSIM 0.869 # 23
Compare
Deblurring REDS VRT Average PSNR 36.79 # 1
Compare
Video Denoising Set8 sigma10 VRT PSNR 37.88 # 2
Compare
Video Denoising Set8 sigma20 VRT PSNR 35.02 # 2
Compare
Video Denoising Set8 sigma30 VRT PSNR 33.35 # 2
Compare
Video Denoising Set8 sigma40 VRT PSNR 32.15 # 3
Compare
Video Denoising Set8 sigma50 VRT PSNR 31.22 # 3
Compare
Video Super-Resolution UDM10 - 4x upscaling VRT PSNR 41.05 # 1
Compare
SSIM 0.9737 # 1
Compare
Video Super-Resolution Vid4 - 4x upscaling VRT PSNR 27.93 # 8
Compare
SSIM 0.8425 # 8
Compare
Video Frame Interpolation Vid4 - 4x upscaling VRT PSNR 27.46 # 1
Compare
SSIM 0.8392 # 1
Compare
Parameters 4450000 # 5
Compare
Video Super-Resolution Vid4 - 4x upscaling - BD degradation VRT PSNR 29.42 # 2
Compare
SSIM 0.8795 # 2
Compare
Space-time Video Super-resolution Vimeo90K-Fast VRT PSNR 36.98 # 1
Compare
SSIM 0.9439 # 1
Compare
Space-time Video Super-resolution Vimeo90K-Medium VRT PSNR 36.01 # 1
Compare
SSIM 0.9434 # 4
Compare

Methods
Add Remove

Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy