Search Results for author:
Found 18 papers, 5 papers with code
Policy Representation via Diffusion Probability Model for Reinforcement Learning
Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration.
Environment Invariant Linear Least Squares
The joint distribution of the response variable and covariate may vary across different environments, yet the conditional expectation of $y$ given the unknown set of important variables are invariant across environments.
Provable Particle-based Primal-Dual Algorithm for Mixed Nash Equilibrium
We consider the general nonconvex nonconcave minimax problem over continuous variables.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto unknown phenomenon that we term \textit{Minority Collapse}, which fundamentally limits the performance of deep learning models on the minority classes.
Mathematical Models of Overparameterized Neural Networks
Deep learning has received considerable empirical successes in recent years.
How to Characterize The Landscape of Overparameterized Convolutional Neural Networks
With the help of a new technique called {\it neural network grafting}, we demonstrate that even during the entire training process, feature distributions of differently initialized networks remain similar at each layer.
Improved Analysis of Clipping Algorithms for Non-convex Optimization
Gradient clipping is commonly used in training deep neural networks partly due to its practicability in relieving the exploding gradient problem.
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks
This new representation overcomes the degenerate situation where all the hidden units essentially have only one meaningful hidden unit in each middle layer, and further leads to a simpler representation of DNNs, for which the training objective can be reformulated as a convex optimization problem via suitable re-parameterization.
Convex Formulation of Overparameterized Deep Neural Networks
This new analysis is consistent with empirical observations that deep neural networks are capable of learning efficient feature representations.
Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
Recently, over-parameterized neural networks have been extensively analyzed in the literature.
A Stochastic Trust Region Method for Non-convex Minimization
We target the problem of finding a local minimum in non-convex finite-sum minimization.
Sharp Analysis for Nonconvex SGD Escaping from Saddle Points
In this paper, we give a sharp analysis for Stochastic Gradient Descent (SGD) and prove that SGD is able to efficiently escape from saddle points and find an $(\epsilon, O(\epsilon^{0. 5}))$-approximate second-order stationary point in $\tilde{O}(\epsilon^{-3. 5})$ stochastic gradient computations for generic nonconvex optimization problems, when the objective function satisfies gradient-Lipschitz, Hessian-Lipschitz, and dispersive noise assumptions.
Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attack
Zeroth-order optimization is an important research topic in machine learning.
SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path-Integrated Differential Estimator
Specially, we prove that the SPIDER-SFO algorithm achieves a gradient computation cost of $\mathcal{O}\left( \min( n^{1/2} \epsilon^{-2}, \epsilon^{-3} ) \right)$ to find an $\epsilon$-approximate first-order stationary point.
Lifted Proximal Operator Machines
LPOM is block multi-convex in all layer-wise weights and activations.
SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
For stochastic first-order method, combining SPIDER with normalized gradient descent, we propose two new algorithms, namely SPIDER-SFO and SPIDER-SFO\textsuperscript{+}, that solve non-convex stochastic optimization problems using stochastic gradients only.
Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation
$O(1/\epsilon)$) convergence rate for non-strongly convex functions, and $O(\sqrt{\kappa}\log(1/\epsilon))$ (v. s.
