XVFI: eXtreme Video Frame Interpolation

ICCV 2021  ·  Hyeonjun Sim, Jihyong Oh, Munchurl Kim ·

In this paper, we firstly present a dataset (X4K1000FPS) of 4K videos of 1000 fps with the extreme motion to the research community for video frame interpolation (VFI), and propose an extreme VFI network, called XVFI-Net, that first handles the VFI for 4K videos with large motion. The XVFI-Net is based on a recursive multi-scale shared structure that consists of two cascaded modules for bidirectional optical flow learning between two input frames (BiOF-I) and for bidirectional optical flow learning from target to input frames (BiOF-T). The optical flows are stably approximated by a complementary flow reversal (CFR) proposed in BiOF-T module. During inference, the BiOF-I module can start at any scale of input while the BiOF-T module only operates at the original input scale so that the inference can be accelerated while maintaining highly accurate VFI performance. Extensive experimental results show that our XVFI-Net can successfully capture the essential information of objects with extremely large motions and complex textures while the state-of-the-art methods exhibit poor performance. Furthermore, our XVFI-Net framework also performs comparably on the previous lower resolution benchmark dataset, which shows a robustness of our algorithm as well. All source codes, pre-trained models, and proposed X4K1000FPS datasets are publicly available at https://github.com/JihyongOh/XVFI.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract


Introduced in the Paper:


Used in the Paper:

Vimeo90K MSU Video Frame Interpolation
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Frame Interpolation MSU Video Frame Interpolation XVFI (S_{tst}=3) Subjective score 1.38 # 2
PSNR 27.35 # 14
SSIM 0.913 # 13
VMAF 63.47 # 14
LPIPS 0.061 # 18
MS-SSIM 0.933 # 13
FPS 5.4 # 2
Video Frame Interpolation MSU Video Frame Interpolation XVFI (S_{tst}=5) PSNR 27.86 # 11
SSIM 0.921 # 7
VMAF 67.25 # 9
LPIPS 0.049 # 13
MS-SSIM 0.955 # 6
Video Frame Interpolation Vimeo90K XVFI PSNR 35.07 # 15
SSIM 0.9760 # 12
Video Frame Interpolation X4K1000FPS XVFI-Net (S_{tst}=5) PSNR 30.12 # 9
SSIM 0.870 # 9
tOF 2.15 # 1
Video Frame Interpolation X4K1000FPS XVFI-Net (S_{tst}=3) PSNR 28.86 # 11
SSIM 0.858 # 10
tOF 2.67 # 2


No methods listed for this paper. Add relevant methods here