Search Results for author:
Found 12 papers, 0 papers with code
On the Convergence of Policy in Unregularized Policy Mirror Descent
In this short note, we give the convergence analysis of the policy in the recent famous policy mirror descent (PMD).
Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer
In this paper, we follow Eftekhari's work to give a non-local convergence analysis of deep linear networks.
Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data
In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss.
Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence
In this paper, we follow Rodomanov and Nesterov’s work to study quasi-Newton methods.
Directional Convergence Analysis under Spherically Symmetric Distribution
We consider the fundamental problem of learning linear predictors (i. e., separable datasets with zero margin) using neural networks with gradient flow or gradient descent.
Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods.
On the Landscape of Sparse Linear Networks
Network pruning, or sparse network has a long history and practical significance in modern applications.
On the Landscape of One-hidden-layer Sparse Networks and Beyond
We show that linear networks can have no spurious valleys under special sparse structures, and non-linear networks could also admit no spurious valleys under a wide final layer.
Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
Quantized Neural Networks (QNNs) use low bit-width fixed-point numbers for representing weight parameters and activations, and are often used in real-world applications due to their saving of computation resources and reproducibility of results.
Towards Understanding the Importance of Noise in Training Neural Networks
Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks.
Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
Stochastic variance-reduced gradient (SVRG) is a classical optimization method.
Hyper-Regularization: An Adaptive Choice for the Learning Rate in Gradient Descent
Specifically, we impose a regularization term on the learning rate via a generalized distance, and cast the joint updating process of the parameter and the learning rate into a maxmin problem.
