Deep Independently Recurrent Neural Network (IndRNN)

11 Oct 2019  ·  Shuai Li, Wanqing Li, Chris Cook, Yanbo Gao ·

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. Due to the better behaved gradient backpropagation, IndRNN with regulated recurrent weights effectively addresses the gradient vanishing and exploding problems and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performance has been achieved on various tasks with IndRNNs compared with the traditional RNN, LSTM and the popular Transformer.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Skeleton Based Action Recognition NTU RGB+D Dense IndRNN Accuracy (CV) 93.97 # 56
Accuracy (CS) 86.70 # 63
Language Modelling Penn Treebank (Character Level) Dense IndRNN Bit per Character (BPC) 1.18 # 8
Language Modelling Penn Treebank (Word Level) Dense IndRNN Test perplexity 56.37 # 27
Language Modelling Penn Treebank (Word Level) Dense IndRNN+dynamic eval Test perplexity 50.97 # 13
Sequential Image Classification Sequential MNIST Dense IndRNN Unpermuted Accuracy 99.48% # 6
Permuted Accuracy 97.2% # 12