In a number of disciplines, the data (e.g., graphs, manifolds) to be analyzed
are non-Euclidean in nature. Geometric deep learning corresponds to techniques
that generalize deep neural network models to such non-Euclidean spaces...
Several recent papers have shown how convolutional neural networks (CNNs) can
be extended to learn with graph-based data. In this work, we study the setting
where the data (or measurements) are ordered, longitudinal or temporal in
nature and live on a Riemannian manifold -- this setting is common in a variety
of problems in statistical machine learning, vision and medical imaging. We
show how recurrent statistical recurrent network models can be defined in such
spaces. We give an efficient algorithm and conduct a rigorous analysis of its
statistical properties. We perform extensive numerical experiments
demonstrating competitive performance with state of the art methods but with
significantly less number of parameters. We also show applications to a
statistical analysis task in brain imaging, a regime where deep neural network
models have only been utilized in limited ways.