Search Results for author:
Found 4 papers, 0 papers with code
Robust Implicit Regularization via Weight Normalization
By analyzing key invariants of the gradient flow and using Lojasiewicz Theorem, we show that weight normalization also has an implicit bias towards sparse solutions in the diagonal linear model, but that in contrast to plain gradient flow, weight normalization enables a robust bias that persists even if the weights are initialized at practically large scale.
More is Less: Inducing Sparsity via Overparameterization
In deep learning it is common to overparameterize neural networks, that is, to use more parameters than training samples.
Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rank
This suggests that deep learning prefers trajectories whose complexity (measuredin terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for thetheoretical understanding of deep learning.
Overparameterization and generalization error: weighted trigonometric interpolation
Motivated by surprisingly good generalization properties of learned deep neural networks in overparameterized scenarios and by the related double descent phenomenon, this paper analyzes the relation between smoothness and low generalization error in an overparameterized linear learning problem.
