Search Results for author:
Found 51 papers, 4 papers with code
On the rates of convergence for learning with convolutional neural networks
Our second result gives new analysis on the covering number of feed-forward neural networks with CNNs as special cases.
Nonlinear functional regression by functional deep neural network with kernel embedding
With the rapid development of deep learning in various fields of science and technology, such as speech recognition, image classification, and natural language processing, recently it is also widely applied in the functional data analysis (FDA) with some empirical success.
Lifting the Veil: Unlocking the Power of Depth in Q-learning
In this paper, we make fundamental contributions to the field of reinforcement learning by answering to the following three questions: Why does deep Q-learning perform so well?
Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos
Data silos, mainly caused by privacy and interoperability, significantly constrain collaborations among different organizations with similar data for the same purpose.
Solving PDEs on Spheres with Physics-Informed Convolutional Neural Networks
In this paper, we establish rigorous analysis of the physics-informed convolutional neural network (PICNN) for solving PDEs on the sphere.
Classification with Deep Neural Networks and Logistic Loss
In this paper, we aim to fill this gap by establishing a novel and elegant oracle-type inequality, which enables us to deal with the boundedness restriction of the target function, and using it to derive sharp convergence rates for fully connected ReLU DNN classifiers trained with logistic loss.
Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning
This paper studies the performance of deep convolutional neural networks (DCNNs) with zero-padding in feature extraction and learning.
Rates of Approximation by ReLU Shallow Neural Networks
We show that ReLU shallow neural networks with $m$ hidden neurons can uniformly approximate functions from the H\"older space $W_\infty^r([-1, 1]^d)$ with rates $O((\log m)^{\frac{1}{2} +d}m^{-\frac{r}{d}\frac{d+2}{d+4}})$ when $r<d/2 +2$.
Learning Theory of Distribution Regression with Neural Networks
In contrast to the classical regression methods, the input variables of distribution regression are probability measures.
Nonparametric regression using over-parameterized shallow ReLU neural networks
It is shown that over-parameterized neural networks can achieve minimax optimal rates of convergence (up to logarithmic factors) for learning functions from certain smooth function classes, if the weights are suitably constrained or regularized.
Optimal Estimates for Pairwise Learning with Deep ReLU Networks
Pairwise learning refers to learning tasks where a loss takes a pair of samples into consideration.
Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks
This sheds light on sufficient or necessary conditions for under-parameterized and over-parameterized NNs trained by GD to attain the desired risk rate of $O(1/\sqrt{n})$.
Distributed Gradient Descent for Functional Learning
In recent years, different types of distributed learning schemes have received increasing attention for their strong advantages in handling large-scale data information.
Approximation of Nonlinear Functionals Using Deep ReLU Networks
In recent years, functional neural networks have been proposed and studied in order to approximate nonlinear continuous functionals defined on $L^p([-1, 1]^s)$ for integers $s\ge1$ and $1\le p<\infty$.
Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression
It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.
Sketching with Spherical Designs for Noisy Data Fitting on Spheres
These interesting findings show that the proposed sketching strategy is capable of fitting massive and noisy data on spheres.
Generalization Analysis for Contrastive Representation Learning
For self-bounding Lipschitz loss functions, we further improve our results by developing optimistic bounds which imply fast rates in a low noise condition.
SignReLU neural network and its approximation ability
Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years.
Approximation analysis of CNNs from a feature extraction view
In this paper we establish some analysis for linear feature extraction by a deep multi-channel convolutional neural networks (CNNs), which demonstrates the power of deep learning over traditional linear transformations, like Fourier, wavelets, redundant dictionary coding methods.
Stability and Generalization for Markov Chain Stochastic Gradient Methods
To the best of our knowledge, this is the first generalization analysis of SGMs when the gradients are sampled from a Markov process.
Differentially Private Stochastic Gradient Descent with Low-Noise
In this paper, we focus on the privacy and utility (measured by excess risk bounds) performances of differentially private stochastic gradient descent (SGD) algorithms in the setting of stochastic convex optimization.
Attention Enables Zero Approximation Error
In this work, we show that with suitable adaptations, the single-head self-attention transformer with a fixed number of transformer encoder blocks and free parameters is able to generate any desired polynomial of the input with no error.
Radial Basis Function Approximation with Distributively Stored Data on Spheres
This paper proposes a distributed weighted regularized least squares algorithm (DWRLS) based on spherical radial basis functions and spherical quadrature rules to tackle spherical data that are stored across numerous local servers and cannot be shared with each other.
Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
In this paper, we study the generalization performance of global minima for implementing empirical risk minimization (ERM) on over-parameterized deep ReLU nets.
Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions
We consider a family of deep neural networks consisting of two groups of convolutional layers, a downsampling operator, and a fully connected layer.
Universal Consistency of Deep Convolutional Neural Networks
Compared with avid research activities of deep convolutional neural networks (DCNNs) in practice, the study of theoretical behaviors of DCNNs lags heavily behind.
Robust Kernel-based Distribution Regression
Regularization schemes for regression have been widely studied in learning theory and inverse problems.
Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints
We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are weaker than those in the literature.
Optimization and Control
Theory of Deep Convolutional Neural Networks II: Spherical Analysis
Deep learning based on deep neural networks of various structures and architectures has been powerful in many practical applications, but it lacks enough theoretical verifications.
Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization
One of the most important challenge of deep learning is to figure out relations between a feature and the depth of deep neural networks (deep nets for short) to reflect the necessity of depth.
Distributed Kernel Ridge Regression with Communications
This paper focuses on generalization performance analysis for distributed algorithms in the framework of learning theory.
Realization of spatial sparseness by deep ReLU nets with massive data
The great success of deep learning poses urgent challenges for understanding its working mechanism and rationality.
Towards Understanding the Spectral Bias of Deep Learning
An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions.
Optimal Stochastic and Online Learning with Individual Iterates
In this paper, we propose a theoretically sound strategy to select an individual iterate of the vanilla SCMD, which is able to achieve optimal rates for both convex and strongly convex problems in a non-smooth learning setting.
Fast Polynomial Kernel Classification for Massive Data
In the era of big data, it is desired to develop efficient machine learning algorithms to tackle massive data challenges such as storage bottleneck, algorithmic scalability, and interpretability.
Distributed filtered hyperinterpolation for noisy data on the sphere
This paper develops distributed filtered hyperinterpolation for noisy data on the sphere, which assigns the data fitting task to multiple servers to find a good approximation of the mapping of input and output data.
Deep Neural Networks for Rotation-Invariance Approximation and Learning
Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high dimensional Euclidian space.
On ADMM in Deep Learning: Convergence and Saturation-Avoidance
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of deep neural networks with sigmoid-type activations (called deep sigmoid nets) over their rectified linear unit (ReLU) counterparts (called deep ReLU nets) in terms of approximation.
Universality of Deep Convolutional Neural Networks
Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, and many other domains.
Construction of neural networks for realization of localized deep learning
The subject of deep learning has recently attracted users of machine learning from various disciplines, including: medical diagnosis and bioinformatics, financial market analysis and online advertisement, speech and handwriting recognition, computer vision and natural language processing, time series forecasting, and search engines.
Convergence of Online Mirror Descent
The condition is $\lim_{t\to\infty}\eta_t=0, \sum_{t=1}^{\infty}\eta_t=\infty$ in the case of positive variances.
Total stability of kernel methods
However, the actually used kernel often depends on one or on a few hyperparameters or the kernel is even data dependent in a much more complicated manner.
Data-dependent Generalization Bounds for Multi-class Classification
In this paper, we study data-dependent generalization error bounds exhibiting a mild dependency on the number of classes, making them suitable for multi-class learning with a large number of label classes.
Distributed learning with regularized least squares
We study distributed learning with the least squares regularization scheme in a reproducing kernel Hilbert space (RKHS).
On the Robustness of Regularized Pairwise Learning Methods Based on Kernels
Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory.
Iterative Regularization for Learning with Convex Loss Functions
We consider the problem of supervised learning with convex loss functions and propose a new form of iterative regularization based on the subgradient method.
Minimax Optimal Rates of Estimation in High Dimensional Additive Models: Universal Phase Transition
We establish minimax optimal rates of convergence for estimation in a high dimensional additive model assuming that it is approximately sparse.
Unregularized Online Learning Algorithms with General Loss Functions
Firstly, we derive explicit convergence rates of the unregularized online learning algorithms for classification associated with a general gamma-activating loss (see Definition 1 in the paper).
Online Pairwise Learning Algorithms with Kernels
In this paper, we study an online algorithm for pairwise learning with a least-square loss function in an unconstrained setting of a reproducing kernel Hilbert space (RKHS), which we refer to as the Online Pairwise lEaRning Algorithm (OPERA).
Consistency Analysis of an Empirical Minimum Error Entropy Algorithm
The error entropy consistency, which requires the error entropy of the learned function to approximate the minimum error entropy, is shown to be always true if the bandwidth parameter tends to 0 at an appropriate rate.
Learning rates for the risk of kernel based quantile regression estimators in additive models
Additive models play an important role in semiparametric statistics.
