Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks

no code implementations • ICML 2018 • Peter Bartlett, Dave Helmbold, Philip Long

We provide polynomial bounds on the number of iterations for gradient descent to approximate the least squares matrix $\Phi$, in the case where the initial hypothesis $\Theta_1 = ... = \Theta_L = I$ has excess loss bounded by a small enough constant.