ConvLSTM is a type of recurrent neural network for spatio-temporal prediction that has convolutional structures in both the input-to-state and state-to-state transitions. The ConvLSTM determines the future state of a certain cell in the grid by the inputs and past states of its local neighbors. This can easily be achieved by using a convolution operator in the state-to-state and input-to-state transitions (see Figure). The key equations of ConvLSTM are shown below, where $∗$ denotes the convolution operator and $\odot$ the Hadamard product:

$$ i_{t} = \sigma\left(W_{xi} ∗ X_{t} + W_{hi} ∗ H_{t−1} + W_{ci} \odot \mathcal{C}_{t−1} + b_{i}\right) $$

$$ f_{t} = \sigma\left(W_{xf} ∗ X_{t} + W_{hf} ∗ H_{t−1} + W_{cf} \odot \mathcal{C}_{t−1} + b_{f}\right) $$

$$ \mathcal{C}_{t} = f_{t} \odot \mathcal{C}_{t−1} + i_{t} \odot \text{tanh}\left(W_{xc} ∗ X_{t} + W_{hc} ∗ \mathcal{H}_{t−1} + b_{c}\right) $$

$$ o_{t} = \sigma\left(W_{xo} ∗ X_{t} + W_{ho} ∗ \mathcal{H}_{t−1} + W_{co} \odot \mathcal{C}_{t} + b_{o}\right) $$

$$ \mathcal{H}_{t} = o_{t} \odot \text{tanh}\left(C_{t}\right) $$

If we view the states as the hidden representations of moving objects, a ConvLSTM with a larger transitional kernel should be able to capture faster motions while one with a smaller kernel can capture slower motions.

To ensure that the states have the same number of rows and same number of columns as the inputs, padding is needed before applying the convolution operation. Here, padding of the hidden states on the boundary points can be viewed as using the state of the outside world for calculation. Usually, before the first input comes, we initialize all the states of the LSTM to zero which corresponds to "total ignorance" of the future.

Source: Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

Latest Papers

PAPER DATE
MCGKT-Net: Multi-level Context Gating Knowledge Transfer Network for Single Image Deraining
Kohei YamamichiXian-Hua Han
2020-10-19
Attention Augmented ConvLSTM forEnvironment Prediction
Bernard LangeMasha ItkinaMykel J. Kochenderfer
2020-10-19
Recurrent convolutional neural network for the surrogate modeling of subsurface flow simulation
Hyung Jun YangTimothy YeoJaewoo An
2020-10-08
RealSmileNet: A Deep End-To-End Network for Spontaneous and Posed Smile Recognition
Yan YangMd Zakir HossainTom GedeonShafin Rahman
2020-10-07
Rain Code : Forecasting Spatiotemporal Precipitation based on Multi-frames Feature using ConvLSTM
Yasuno TakatoIshii AkiraAmakata Masazumi
2020-09-30
Application of LSTM architectures for next frame forecasting in Sentinel-1 images time series
Waytehad MoskolaïWahabou AbdouAlbert DipandaDina Taiwe Kolyang
2020-09-02
DeepSTCL: A Deep Spatio-temporal ConvLSTM for Travel Demand Prediction
Dongjie WangYan YangShangming Ning
2020-08-22
Physics-Informed Deep Neural Networks for Transient Electromagnetic Analysis
Oameed NoakoasteenShu WangZhen PengChristos Christodoulou
2020-08-04
Physics-informed Tensor-train ConvLSTM for Volumetric Velocity Forecasting
Yu HuangYufei TangHanqi ZhuangJames VanZwietenLaurent Cherubin
2020-08-04
Frame-To-Frame Consistent Semantic Segmentation
Manuel RebolPatrick Knöbelreiter
2020-08-03
SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction
Sriram N NBuyu LiuFrancesco PittalugaManmohan Chandraker
2020-07-26
Alternating ConvLSTM: Learning Force Propagation with Alternate State Updates
Congyue DengTai-Jiang MuShi-Min Hu
2020-06-14
Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
Adrian Sanchez-CaballeroDavid Fuentes-JimenezCristina Losada-Gutiérrez
2020-06-13
Open-Narrow-Synechiae Anterior Chamber Angle Classification in AS-OCT Sequences
Huaying HaoHuazhu FuYanwu XuJianlong YangFei LiXiulan ZhangJiang LiuYitian Zhao
2020-06-09
Mosaic Super-resolution via Sequential Feature Pyramid Networks
Mehrdad ShoeibyMohammad Ali ArminSadegh AliakbarianSaeed AnwarLars Petersson
2020-04-15
Multi-level Context Gating of Embedded Collective Knowledge for Medical Image Segmentation
| Maryam Asadi-AghbolaghiReza AzadMahmood FathySergio Escalera
2020-03-10
Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
| Xiaoyu XiangYapeng TianYulun ZhangYun FuJan P. AllebachChenliang Xu
2020-02-26
Dual Convolutional LSTM Network for Referring Image Segmentation
Linwei YeZhi LiuYang Wang
2020-01-30
Don't Forget The Past: Recurrent Depth Estimation from Monocular Video
Vaishakh PatilWouter Van GansbekeDengxin DaiLuc Van Gool
2020-01-08
Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Anonymous
2020-01-01
Learned Video Compression via Joint Spatial-Temporal Correlation Exploration
Haojie LiuHan shenLichao HuangMing LuTong ChenZhan Ma
2019-12-13
Spatiotemporal deep learning model for citywide air pollution interpolation and prediction
Van-Duc LeTien-Cuong BuiSang Kyun Cha
2019-11-29
Non-Local ConvLSTM for Video Compression Artifact Reduction
Yi XuLongwen GaoKai TianShuigeng ZhouHuyang Sun
2019-10-27
Utilizing Temporal Information in Deep Convolutional Network for Efficient Soccer Ball Detection and Tracking
| Anna KuklevaMohammad Asif KhanHafez FaraziSven Behnke
2019-09-05
Dynamic Spatial-Temporal Representation Learning for Traffic Flow Prediction
| Lingbo LiuJiajie ZhenGuanbin LiGeng ZhanZhaocheng HeBowen DuLiang Lin
2019-09-02
Bi-Directional ConvLSTM U-Net with Densley Connected Convolutions
| Reza AzadMaryam Asadi-AghbolaghiMahmood FathySergio Escalera
2019-08-31
Multi-Grained Spatio-temporal Modeling for Lip-reading
Chenhao Wang
2019-08-30
An End-to-end Video Text Detector with Online Tracking
Hongyuan YuChengquan ZhangXuan LiJunyu HanErrui DingLiang Wang
2019-08-20
Separable Convolutional LSTMs for Faster Video Segmentation
| Andreas PfeufferKlaus Dietmayer
2019-07-16
Simple vs complex temporal recurrences for video saliency prediction
| Panagiotis LinardosEva MohedanoJuan Jose NietoNoel E. O'ConnorXavier Giro-i-NietoKevin McGuinness
2019-07-03
FORECAST-CLSTM: A New Convolutional LSTM Network for Cloudage Nowcasting
Chao TanXin FengJianwu LongLi Geng
2019-05-19
Feature Extraction and Classification Based on Spatial-Spectral ConvLSTM Neural Network for Hyperspectral Images
Wen-Shuai HuHeng-Chao LiLei PanWei LiRan TaoQian Du
2019-05-09
Frame-Recurrent Video Inpainting by Robust Optical Flow Inference
Yifan DingChuan WangHaibin HuangJiaming LiuJue WangLiqiang Wang
2019-05-08
FACLSTM: ConvLSTM with Focused Attention for Scene Text Recognition
Qingqing WangWenjing JiaXiangjian HeYue LuMichael BlumensteinYe Huang
2019-04-20
Quality-Gated Convolutional LSTM for Enhancing Compressed Video
| Ren YangXiaoyan SunMai XuWenjun Zeng
2019-03-11
Spatio-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data
Ling ZhangLe LuXiaosong WangRobert M. ZhuMohammadhadi BagheriRonald M. SummersJianhua Yao
2019-02-23
Predicting tongue motion in unlabeled ultrasound videos using convolutional LSTM neural network
| Chaojie ZhaoPeng ZhangJian ZhuChengrui WuHuaimin WangKele Xu
2019-02-19
Grids versus Graphs: Partitioning Space for Improved Taxi Demand-Supply Forecasts
Neema DavisGaurav RainaKrishna Jagannathan
2019-02-18
Attention-driven Tree-structured Convolutional LSTM for High Dimensional Data Understanding
Bin KongXin WangJunjie BaiYi LuFeng GaoKunlin CaoQi SongShaoting ZhangSiwei LyuYoubing Yin
2019-01-29
Feature Boosting Network For 3D Pose Estimation
Jun LiuHenghui DingAmir ShahroudyLing-Yu DuanXudong JiangGang WangAlex C. Kot
2019-01-15
Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos
Chinedu Innocent NwoyeDidier MutterJacques MarescauxNicolas Padoy
2018-12-04
Attention in Convolutional LSTM for Gesture Recognition
| Liang ZhangGuangming ZhuLin MeiPeiyi ShenSyed Afaq Ali ShahMohammed Bennamoun
2018-12-01
A multi-level convolutional LSTM model for the segmentation of left ventricle myocardium in infarcted porcine cine MR images
Dongqing ZhangIlknur IckeBelma DogdasSarayu ParimalSmita SampathJoseph ForbesAnsuman BagchiChih-Liang ChinAntong Chen
2018-11-14
Attentive Crowd Flow Machines
Lingbo LiuRuimao ZhangJiefeng PengGuanbin LiBowen DuLiang Lin
2018-09-01
Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection
| Hongmei SongWenguan WangSanyuan ZhaoJianbing ShenKin-Man Lam
2018-09-01
Future Semantic Segmentation with Convolutional LSTM
Seyed shahabeddin NabaviMrigank RochanYangWang
2018-07-20
STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection
Sangmin LeeHak Gu KimYong Man Ro
2018-04-23
Temporally Identity-Aware SSD with Attentional LSTM
| Xingyu ChenJunzhi YuZhengxing Wu
2018-03-01
Effective Quantization Approaches for Recurrent Neural Networks
Md Zahangir AlomAdam T MoodyNaoya MaruyamaBrian C Van EssenTarek M. Taha
2018-02-07
DeepRain: ConvLSTM Network for Precipitation Prediction using Multichannel Radar Data
Seongchan KimSeungkyun HongMinsu JohSa-kwang Song
2017-11-07
Spatiotemporal Modeling for Crowd Counting in Videos
Feng XiongXingjian ShiDit-Yan Yeung
2017-07-25
Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
| Xingjian ShiZhihan GaoLeonard LausenHao WangDit-Yan YeungWai-kin WongWang-chun Woo
2017-06-12
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
| Xingjian ShiZhourong ChenHao WangDit-Yan YeungWai-kin WongWang-chun Woo
2015-06-13

Categories