CDFI: Compression-Driven Network Design for Frame Interpolation

CVPR 2021  ยท  Tianyu Ding, Luming Liang, Zhihui Zhu, Ilya Zharkov ยท

DNN-based frame interpolation--that generates the intermediate frames given two consecutive frames--typically relies on heavy model architectures with a huge number of features, preventing them from being deployed on systems with limited resources, e.g., mobile devices. We propose a compression-driven network design for frame interpolation (CDFI), that leverages model pruning through sparsity-inducing optimization to significantly reduce the model size while achieving superior performance. Concretely, we first compress the recently proposed AdaCoF model and show that a 10X compressed AdaCoF performs similarly as its original counterpart; then we further improve this compressed model by introducing a multi-resolution warping module, which boosts visual consistencies with multi-level details. As a consequence, we achieve a significant performance gain with only a quarter in size compared with the original AdaCoF. Moreover, our model performs favorably against other state-of-the-arts in a broad range of datasets. Finally, the proposed compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithm. Our source code is available at https://github.com/tding1/CDFI.

PDF Abstract CVPR 2021 PDF CVPR 2021 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Frame Interpolation Middlebury CDFI SSIM 0.966 # 2
PSNR 37.14 # 4
LPIPS 0.007 # 1
Video Frame Interpolation MSU Video Frame Interpolation CDFI PSNR 26.99 # 16
SSIM 0.908 # 15
VMAF 61.72 # 16
LPIPS 0.051 # 15
MS-SSIM 0.926 # 16
Video Frame Interpolation UCF101 CDFI PSNR 35.21 # 12
LPIPS 0.015 # 1
Video Frame Interpolation Vimeo90K CDFI PSNR 35.17 # 14
LPIPS 0.010 # 1

Methods