Video Prediction

183 papers with code • 20 benchmarks • 25 datasets

Video Prediction is the task of predicting future frames given past video frames.

Gif credit: MAGVIT

Source: Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames

Libraries

Use these libraries to find Video Prediction models and implementations

Most implemented papers

Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics

Yunbo426/MIM CVPR 2019

Natural spatiotemporal processes can be highly non-stationary in many ways, e. g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting.

Scalable Gradients for Stochastic Differential Equations

google-research/torchsde 5 Jan 2020

The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations.

Unsupervised Learning for Physical Interaction through Video Prediction

tensorflow/models NeurIPS 2016

A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment.

Video Frame Synthesis using Deep Voxel Flow

liuziwei7/voxel-flow ICCV 2017

We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow.

Self-Supervised Visual Planning with Temporal Skip Connections

CompVis/image2video-synthesis-using-cINNs 15 Oct 2017

One learning signal that is always available for autonomously collected data is prediction: if a robot can learn to predict the future, it can use this predictive model to take actions to produce desired outcomes, such as moving an object to a particular location.

Stochastic Variational Video Prediction

StanfordVL/roboturk_real_dataset ICLR 2018

We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods.

Stochastic Video Generation with a Learned Prior

edenton/svg ICML 2018

Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.

Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning

febert/neuralproposal_cem_classproject 6 Oct 2018

We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and non-prehensile manipulation.

Towards Accurate Generative Models of Video: A New Metric & Challenges

wilson1yan/VideoGPT 3 Dec 2018

To this extent we propose Fr\'{e}chet Video Distance (FVD), a new metric for generative models of video, and StarCraft 2 Videos (SCV), a benchmark of game play from custom starcraft 2 scenarios that challenge the current capabilities of generative models of video.

Eidetic 3D LSTM: A Model for Video Prediction and Beyond

chengtan9907/simvpv2 ICLR 2019

We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance.