Search Results for author:
Found 18 papers, 5 papers with code
Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion
We answer this question in the affirmative by giving a particular construction of an Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded gradients at any depth.
On the impact of activation and normalization in obtaining isometric embeddings at initialization
In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs.
Efficient displacement convex optimization with particle gradient descent
Particle gradient descent, which uses particles to represent a probability measure and performs gradient descent on particles in parallel, is widely used to optimize functions of probability measures.
On Bridging the Gap between Mean Field and Finite Width in Deep Random Neural Networks with Batch Normalization
Mean field theory is widely used in the theoretical studies of neural networks.
Polynomial-time Sparse Measure Recovery: From Mean Field Theory to Algorithm Design
Mean field theory has provided theoretical insights into various algorithms by letting the problem size tend to infinity.
Rethinking the Variational Interpretation of Accelerated Optimization Methods
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization.
Batch Normalization Orthogonalizes Representations in Deep Random Networks
This paper underlines a subtle property of batch-normalization (BN): Successive batch normalizations with random linear transformations make hidden representations increasingly orthogonal across layers of a deep neural network.
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order optimizers.
Batch normalization provably avoids ranks collapse for randomly initialised deep networks
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used.
Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used.
Mixing of Stochastic Accelerated Gradient Descent
We study the mixing properties for stochastic accelerated gradient descent (SAGD) on least-squares regression.
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks.
Local Saddle Point Optimization: A Curvature Exploitation Approach
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems.
Escaping Saddles with Stochastic Gradients
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions.
Accelerated Dual Learning by Homotopic Initialization
Gradient descent and coordinate descent are well understood in terms of their asymptotic behavior, but less so in a transient regime often used for approximations in machine learning.
DynaNewton - Accelerating Newton's Method for Machine Learning
Solutions on this path are tracked such that the minimizer of the previous objective is guaranteed to be within the quadratic convergence region of the next objective to be optimized.
Starting Small -- Learning with Adaptive Sample Sizes
For many machine learning problems, data is abundant and it may be prohibitive to make multiple passes through the full training set.
Estimating Diffusion Network Structures: Recovery Conditions, Sample Complexity & Soft-thresholding Algorithm
Can we recover the hidden network structures from these observed cascades?
