Search Results for author:
Found 12 papers, 6 papers with code
Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks
In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization.
The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks
We introduce the class of quasi-homogeneous models, which is expressive enough to describe nearly all neural networks with homogeneous activations, even those with biases, residual connections, and normalization layers, while structured enough to enable geometric analysis of its gradient dynamics.
Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes
Numerically, we observe that neural network loss functions possesses a multiscale structure, manifested in two ways: (1) in a neighborhood of minima, the loss mixes a continuum of scales and grows subquadratically, and (2) in a larger region, the loss shows several separate scales clearly.
Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks
In nature, symmetry governs regularities, while symmetry breaking brings texture.
Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations, and anomalous diffusion
In this work we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD).
The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
In this work we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD).
Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
In nature, symmetry governs regularities, while symmetry breaking brings texture.
Symmetry, Conservation Laws, and Learning Dynamics in Neural Networks
Overall, by exploiting symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
Overall, by exploiting symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.
Pruning neural networks without any data by iteratively conserving synaptic flow
Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time.
Two Routes to Scalable Credit Assignment without Weight Symmetry
The neural plausibility of backpropagation has long been disputed, primarily for its use of non-local weight transport $-$ the biologically dubious requirement that one neuron instantaneously measure the synaptic weights of another.
Loss Landscapes of Regularized Linear Autoencoders
Autoencoders are a deep learning model for representation learning.
