As an efficient recurrent neural network (RNN) model, reservoir computing
(RC) models, such as Echo State Networks, have attracted widespread attention
in the last decade. However, while they have had great success with time series
data , , many time series have a multiscale structure, which a
single-hidden-layer RC model may have difficulty capturing. In this paper, we
propose a novel hierarchical reservoir computing framework we call Deep Echo
State Networks (Deep-ESNs). The most distinctive feature of a Deep-ESN is its
ability to deal with time series through hierarchical projections.
Specifically, when an input time series is projected into the high-dimensional
echo-state space of a reservoir, a subsequent encoding layer (e.g., a PCA,
autoencoder, or a random projection) can project the echo-state representations
into a lower-dimensional space. These low-dimensional representations can then
be processed by another ESN. By using projection layers and encoding layers
alternately in the hierarchical framework, a Deep-ESN can not only attenuate
the effects of the collinearity problem in ESNs, but also fully take advantage
of the temporal kernel property of ESNs to explore multiscale dynamics of time
series. To fuse the multiscale representations obtained by each reservoir, we
add connections from each encoding layer to the last output layer. Theoretical
analyses prove that stability of a Deep-ESN is guaranteed by the echo state
property (ESP), and the time complexity is equivalent to a conventional ESN.
Experimental results on some artificial and real world time series demonstrate
that Deep-ESNs can capture multiscale dynamics, and outperform both standard
ESNs and previous hierarchical ESN-based models.