Sampling-based Gradient Regularization for Capturing Long-Term Dependencies in Recurrent Neural Networks

no code implementations • 24 Jun 2016 • Artem Chernodub, Dimitri Nowicki

In this paper we propose a novel universal technique that makes the norm of the gradient stay in the suitable range.

Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

no code implementations • 8 Apr 2016 • Artem Chernodub, Dimitri Nowicki

We propose a novel activation function that implements piece-wise orthogonal non-linear mappings based on permutations.