Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics

CVPR 2019 Yunbo WangJianjin ZhangHongyu ZhuMingsheng LongJianmin WangPhilip S Yu

Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term... (read more)

PDF Abstract

Evaluation results from the paper

Task Dataset Model Metric name Metric value Global rank Compare
Video Prediction Human3.6M MIM SSIM 0.790 # 1
Video Prediction Human3.6M MIM MSE 429.9 # 1
Video Prediction Human3.6M MIM MAE 1782.8 # 1