Search Results for author:
Found 26 papers, 2 papers with code
Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction
Despite the generality of our results, we achieve an improved convergence rate in both the SGD and SVRG settings when specialized to the standard Langevin dynamics.
Tight and fast generalization error bound of graph embedding in metric space
However, recent theoretical analyses have shown a much higher upper bound on non-Euclidean graph embedding's generalization error than Euclidean one's, where a high generalization error indicates that the incompleteness and noise in the data can significantly damage learning performance.
Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems
The entropic fictitious play (EFP) is a recently proposed algorithm that minimizes the sum of a convex functional and entropy in the space of measures -- such an objective naturally arises in the optimization of a two-layer neural network in the mean-field regime.
Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions
Stochastic gradient descent is a workhorse for training deep neural networks due to its excellent generalization performance.
Koopman-based generalization bound: New aspect for full-rank weights
Our bound is tighter than existing norm-based bounds when the condition numbers of weight matrices are small.
Convex Analysis of the Mean Field Langevin Dynamics
In this work, we give a concise and self-contained convergence rate analysis of the mean field Langevin dynamics with respect to the (regularized) objective function in both continuous and discrete time settings.
Generalization Bounds for Graph Embedding Using Negative Sampling: Linear vs Hyperbolic
Graph embedding, which represents real-world entities in a mathematical space, has enabled numerous applications such as analyzing natural languages, social networks, biochemical networks, and knowledge bases. It has been experimentally shown that graph embedding in hyperbolic space can represent hierarchical tree-like data more effectively than embedding in linear space, owing to hyperbolic space's exponential growth property.
Particle Stochastic Dual Coordinate Ascent: Exponential convergent algorithm for mean field neural network optimization
We introduce Particle-SDCA, a gradient-based optimization algorithm for two-layer neural networks in the mean field regime that achieves exponential convergence rate in regularized empirical risk minimization.
Generalization Error Bound for Hyperbolic Ordinal Embedding
Hyperbolic ordinal embedding (HOE) represents entities as points in hyperbolic space so that they agree as well as possible with given constraints in the form of entity i is more similar to entity j than to entity k. It has been experimentally shown that HOE can obtain representations of hierarchical data such as a knowledge base and a citation network effectively, owing to hyperbolic space's exponential growth property.
Particle Dual Averaging: Optimization of Mean Field Neural Network with Global Convergence Rate Analysis
An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain.
BODAME: Bilevel Optimization for Defense Against Model Extraction
Model extraction attacks have become serious issues for service providers using machine learning.
Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain.
A Novel Global Spatial Attention Mechanism in Convolutional Neural Network for Medical Image Classification
In this paper, we thus propose a novel global spatial attention mechanism in CNNs mainly for medical image classification.
Online Robust and Adaptive Learning from Data Streams
We address the relation between the two parameters: one is the step size of the stochastic approximation, and the other is the threshold parameter of the norm of the stochastic update.
Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
In this study, we show that the averaged stochastic gradient descent can achieve the minimax optimal convergence rate, with the global convergence guarantee, by exploiting the complexities of the target function and the RKHS associated with the NTK.
When Does Preconditioning Help or Hurt Generalization?
While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question.
Exponential Convergence Rates of Classification Errors on Learning with SGD and Random Features
To address this problem, sketching and stochastic gradient methods are the most commonly used techniques to derive efficient large-scale learning algorithms.
Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space
The results show that deep learning has better dependence on the input dimensionality if the target function possesses anisotropic smoothness, and it achieves an adaptive rate for functions with spatially inhomogeneous smoothness.
Data Cleansing for Models Trained with SGD
Data cleansing is a typical approach used to improve the accuracy of machine learning models, which, however, requires extensive domain knowledge to identify the influential instances that affect the models.
Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
Most studies especially focused on the regression problems with the squared loss function, except for a few, and the importance of the positivity of the neural tangent kernel has been pointed out.
Stochastic Gradient Descent with Exponential Convergence Rates of Expected Classification Errors
In this paper, we show an exponential convergence of the expected classification error in the final phase of the stochastic gradient descent for a wide class of differentiable convex loss functions under similar assumptions.
Functional Gradient Boosting based on Residual Network Perception
Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well.
Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models
In this paper, this phenomenon is explained from the functional gradient method perspective of the gradient layer.
Stochastic Particle Gradient Descent for Infinite Ensembles
The superior performance of ensemble methods with infinite models are well known.
Accelerated Stochastic Gradient Descent for Minimizing Finite Sums
We propose an optimization method for minimizing the finite sums of smooth convex functions.
Stochastic Proximal Gradient Descent with Acceleration Techniques
Accelerated proximal gradient descent (APG) and proximal stochastic variance reduction gradient (Prox-SVRG) are in a trade-off relationship.
