Search Results for author:
Found 18 papers, 5 papers with code
SGD with memory: fundamental properties and stochastic acceleration
In the non-stochastic setting, the optimal exponent $\xi$ in the loss convergence $L_t\sim C_Lt^{-\xi}$ is double that in plain GD and is achievable using Heavy Ball (HB) with a suitable schedule; this no longer works in the presence of mini-batch noise.
Generalization error of spectral algorithms
In the present work, we consider the training of kernels with a family of $\textit{spectral algorithms}$ specified by profile $h(\lambda)$, and including KRR and GD as special cases.
Learnability of high-dimensional targets by two-parameter models and gradient flow
We explore the theoretical possibility of learning $d$-dimensional targets with $W$-parameter models by gradient flow (GF) when $W<d$.
A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta
Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models.
Embedded Ensembles: Infinite Width Limit and Operating Regimes
In this limit, we identify two ensemble regimes - independent and collective - depending on the architecture and initialization strategy of ensemble models.
Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions
In this paper, we propose a new spectral condition providing tighter upper bounds for problems with power law optimization trajectories.
Explicit loss asymptotics in the gradient descent training of neural networks
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss values.
Universal scaling laws in the gradient descent training of neural networks
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss values.
Elementary superexpressive activations
We call a finite family of activation functions superexpressive if any multivariate continuous function can be approximated by a neural network that uses these activations and has a fixed architecture only depending on the number of input variables (i. e., to achieve any accuracy we only need to adjust the weights, without increasing the number of neurons).
Low-loss connection of weight vectors: distribution-based approaches
Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately.
The phase diagram of approximation rates for deep neural networks
We explore the phase diagram of approximation rates for deep neural networks and prove several new theoretical results.
Collective evolution of weights in wide neural networks
We test our general method in the special case of linear free-knot splines, and find good agreement between theory and experiment in observations of global optima, stability of stationary points, and convergence rates.
Universal approximations of invariant maps by neural networks
We prove this model to be a universal approximator for continuous SE(2)--equivariant signal transformations.
Optimal approximation of continuous functions by very deep ReLU networks
We consider approximations of general continuous functions on finite-dimensional cubes by general deep ReLU neural networks and study the approximation rates with respect to the modulus of continuity of the function and the total number of weights $W$ in the network.
Quantified advantage of discontinuous weight selection in approximations with deep neural networks
We consider approximations of 1D Lipschitz functions by deep ReLU networks of a fixed width.
Geometric features for voxel-based surface recognition
We introduce a library of geometric voxel features for CAD surface recognition/retrieval tasks.
Error bounds for approximations with deep ReLU networks
We study expressive power of shallow and deep neural networks with piece-wise linear activation functions.
GTApprox: surrogate modeling for industrial design
We describe GTApprox - a new tool for medium-scale surrogate modeling in industrial design.
