A Dynamic Multi-Scale Voxel Flow Network for Video Prediction

The performance of video prediction has been greatly boosted by advanced deep neural networks. However, most of the current methods suffer from large model sizes and require extra inputs, e.g., semantic/depth maps, for promising performance. For efficiency consideration, in this paper, we propose a Dynamic Multi-scale Voxel Flow Network (DMVFN) to achieve better video prediction performance at lower computational costs with only RGB images, than previous methods. The core of our DMVFN is a differentiable routing module that can effectively perceive the motion scales of video frames. Once trained, our DMVFN selects adaptive sub-networks for different inputs at the inference stage. Experiments on several benchmarks demonstrate that our DMVFN is an order of magnitude faster than Deep Voxel Flow and surpasses the state-of-the-art iterative-based OPT on generated image quality. Our code and demo are available at https://huxiaotaostasy.github.io/DMVFN/.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Prediction Cityscapes DMVFN MS-SSIM 0.9573 # 1
LPIPS 0.0558 # 1
Video Prediction DAVIS 2017 DMVFN MS-SSIM 0.8397 # 1
LPIPS 0.0996 # 1
Video Prediction KITTI DMVFN MS-SSIM 0.8853 # 1
LPIPS 0.1074 # 1
Video Prediction Vimeo90K DMVFN MS-SSIM 0.9701 # 1
LPIPS 0.0369 # 2