Search Results for author:
Found 7 papers, 3 papers with code
Should Under-parameterized Student Networks Copy or Average Teacher Weights?
Approximating $f^*$ with a neural network with $n< k$ neurons can thus be seen as fitting an under-parameterized "student" network with $n$ neurons to a "teacher" network with $k$ neurons.
Statistical physics, Bayesian inference and neural information processing
Lecture notes from the course given by Professor Sara A. Solla at the Les Houches summer school on "Statistical physics of Machine Learning".
MLPGradientFlow: going with the flow of multilayer perceptrons (and finding minima fast and accurately)
MLPGradientFlow is a software package to solve numerically the gradient flow differential equation $\dot \theta = -\nabla \mathcal L(\theta; \mathcal D)$, where $\theta$ are the parameters of a multi-layer perceptron, $\mathcal D$ is some data set, and $\nabla \mathcal L$ is the gradient of a loss function.
Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity
The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance $\sigma^2$ of the parameters at initialization $\theta_0$.
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
For a two-layer overparameterized network of width $ r^*+ h =: m $ we explicitly describe the manifold of global minima: it consists of $ T(r^*, m) $ affine subspaces of dimension at least $ h $ that are connected to one another.
Kernel Alignment Risk Estimator: Risk Prediction from Training Data
Under a natural universality assumption (that the KRR moments depend asymptotically on the first two moments of the observations) we capture the mean and variance of the KRR predictor.
Implicit Regularization of Random Feature Models
We investigate, by means of random matrix theory, the connection between Gaussian RF models and Kernel Ridge Regression (KRR).
