XVFI: eXtreme Video Frame Interpolation

ICCV 2021  ·  Hyeonjun Sim, Jihyong Oh, Munchurl Kim ·

In this paper, we firstly present a dataset (X4K1000FPS) of 4K videos of 1000 fps with the extreme motion to the research community for video frame interpolation (VFI), and propose an extreme VFI network, called XVFI-Net, that first handles the VFI for 4K videos with large motion. The XVFI-Net is based on a recursive multi-scale shared structure that consists of two cascaded modules for bidirectional optical flow learning between two input frames (BiOF-I) and for bidirectional optical flow learning from target to input frames (BiOF-T). The optical flows are stably approximated by a complementary flow reversal (CFR) proposed in BiOF-T module. During inference, the BiOF-I module can start at any scale of input while the BiOF-T module only operates at the original input scale so that the inference can be accelerated while maintaining highly accurate VFI performance. Extensive experimental results show that our XVFI-Net can successfully capture the essential information of objects with extremely large motions and complex textures while the state-of-the-art methods exhibit poor performance. Furthermore, our XVFI-Net framework also performs comparably on the previous lower resolution benchmark dataset, which shows a robustness of our algorithm as well. All source codes, pre-trained models, and proposed X4K1000FPS datasets are publicly available at https://github.com/JihyongOh/XVFI.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract

Datasets


Introduced in the Paper:

X4K1000FPS

Used in the Paper:

Vimeo90K
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Frame Interpolation Vimeo90K XVFI PSNR 35.07 # 10
SSIM 0.9760 # 8
Video Frame Interpolation X4K1000FPS XVFI-Net (S_{tst}=5) PSNR 30.12 # 4
SSIM 0.870 # 4
tOF 2.15 # 1
Video Frame Interpolation X4K1000FPS XVFI-Net (S_{tst}=3) PSNR 28.86 # 5
SSIM 0.858 # 5
tOF 2.67 # 2

Methods


No methods listed for this paper. Add relevant methods here