Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

no code implementations • 24 Sep 2018 • Wojciech Tarnowski, Piotr Warchoł, Stanisław Jastrzębski, Jacek Tabor, Maciej A. Nowak

We propose that in ResNets this can be resolved based on our results, by ensuring the same level of dynamical isometry at initialization.