# Implicit bias of gradient descent for mean squared error regression with wide neural networks

12 Jun 2020Hui JinGuido Montúfar

We investigate gradient descent training of wide neural networks and the corresponding implicit bias in function space. Focusing on 1D regression, we show that the solution of training a width-$n$ shallow ReLU network is within $n^{- 1/2}$ of the function which fits the training data and whose difference from initialization has smallest 2-norm of the second derivative weighted by $1/\zeta$... (read more)

PDF Abstract