Overcoming the vanishing gradient problem in plain recurrent networks

no code implementations • ICLR 2018 • Yuhuang Hu, Adrian Huber, Jithendar Anumula, Shih-Chii Liu

Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs.

Permuted-MNIST Question Answering