Generating Videos with Scene Dynamics

We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.

PDF Abstract NeurIPS 2016 PDF NeurIPS 2016 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Self-Supervised Action Recognition UCF101 VideoGan (C3D) 3-fold Accuracy 52.1 # 51
Pre-Training Dataset UCF101 # 1
Frozen false # 1
Video Generation UCF-101 16 frames, 64x64, Unconditional VGAN Inception Score 8.18 # 7
Video Generation UCF-101 16 frames, Unconditional, Single GPU VGAN Inception Score 8.18 # 7

Methods


No methods listed for this paper. Add relevant methods here