Unitary Evolution Recurrent Neural Networks

20 Nov 2015Martin ArjovskyAmar ShahYoshua Bengio

Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies... (read more)

PDF Abstract

Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Sequential Image Classification Sequential MNIST LSTM Unpermuted Accuracy 98.2% # 4
Sequential Image Classification Sequential MNIST LSTM Permuted Accuracy 88% # 6