Video Enhancement with Task-Oriented Flow

24 Nov 2017  ·  Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, William T. Freeman ·

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

PDF Abstract


Introduced in the Paper:


Used in the Paper:

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Frame Interpolation Middlebury ToFlow Interpolation Error 5.49 # 7
Video Super-Resolution Vid4 - 4x upscaling - BD degradation TOFlow PSNR 25.85 # 18
SSIM 0.7659 # 18
Video Frame Interpolation Vimeo90K ToFlow PSNR 33.73 # 24


No methods listed for this paper. Add relevant methods here