# A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

18 Jan 2019Umut SimsekliLevent SagunMert Gurbuzbalaban

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion... (read more)

PDF Abstract