Search Results for author:
Found 30 papers, 2 papers with code
Attention Enables Zero Approximation Error
In this work, we show that with suitable adaptations, the single-head self-attention transformer with a fixed number of transformer encoder blocks and free parameters is able to generate any desired polynomial of the input with no error.
Radial Basis Function Approximation with Distributively Stored Data on Spheres
This paper proposes a distributed weighted regularized least squares algorithm (DWRLS) based on spherical radial basis functions and spherical quadrature rules to tackle spherical data that are stored across numerous local servers and cannot be shared with each other.
Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
In this paper, we study the generalization performance of global minima for implementing empirical risk minimization (ERM) on over-parameterized deep ReLU nets.
Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions
We consider a family of deep neural networks consisting of two groups of convolutional layers, a downsampling operator, and a fully connected layer.
Universal Consistency of Deep Convolutional Neural Networks
Compared with avid research activities of deep convolutional neural networks (DCNNs) in practice, the study of theoretical behaviors of DCNNs lags heavily behind.
Robust Kernel-based Distribution Regression
Regularization schemes for regression have been widely studied in learning theory and inverse problems.
Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints
We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are weaker than those in the literature.
Optimization and Control
Theory of Deep Convolutional Neural Networks II: Spherical Analysis
Deep learning based on deep neural networks of various structures and architectures has been powerful in many practical applications, but it lacks enough theoretical verifications.
Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization
One of the most important challenge of deep learning is to figure out relations between a feature and the depth of deep neural networks (deep nets for short) to reflect the necessity of depth.
Distributed Kernel Ridge Regression with Communications
This paper focuses on generalization performance analysis for distributed algorithms in the framework of learning theory.
Realization of spatial sparseness by deep ReLU nets with massive data
The great success of deep learning poses urgent challenges for understanding its working mechanism and rationality.
Towards Understanding the Spectral Bias of Deep Learning
An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions.
Optimal Stochastic and Online Learning with Individual Iterates
In this paper, we propose a theoretically sound strategy to select an individual iterate of the vanilla SCMD, which is able to achieve optimal rates for both convex and strongly convex problems in a non-smooth learning setting.
Fast Polynomial Kernel Classification for Massive Data
In the era of big data, it is highly desired to develop efficient machine learning algorithms to tackle massive data challenges such as storage bottleneck, algorithmic scalability, and interpretability.
Distributed filtered hyperinterpolation for noisy data on the sphere
This paper develops distributed filtered hyperinterpolation for noisy data on the sphere, which assigns the data fitting task to multiple servers to find a good approximation of the mapping of input and output data.
Deep Neural Networks for Rotation-Invariance Approximation and Learning
Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high dimensional Euclidian space.
On ADMM in Deep Learning: Convergence and Saturation-Avoidance
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of deep neural networks with sigmoid-type activations (called deep sigmoid nets) over their rectified linear unit (ReLU) counterparts (called deep ReLU nets) in terms of approximation.
Universality of Deep Convolutional Neural Networks
Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, and many other domains.
Construction of neural networks for realization of localized deep learning
The subject of deep learning has recently attracted users of machine learning from various disciplines, including: medical diagnosis and bioinformatics, financial market analysis and online advertisement, speech and handwriting recognition, computer vision and natural language processing, time series forecasting, and search engines.
Convergence of Online Mirror Descent
The condition is $\lim_{t\to\infty}\eta_t=0, \sum_{t=1}^{\infty}\eta_t=\infty$ in the case of positive variances.
Total stability of kernel methods
However, the actually used kernel often depends on one or on a few hyperparameters or the kernel is even data dependent in a much more complicated manner.
Data-dependent Generalization Bounds for Multi-class Classification
In this paper, we study data-dependent generalization error bounds exhibiting a mild dependency on the number of classes, making them suitable for multi-class learning with a large number of label classes.
Distributed learning with regularized least squares
We study distributed learning with the least squares regularization scheme in a reproducing kernel Hilbert space (RKHS).
On the Robustness of Regularized Pairwise Learning Methods Based on Kernels
Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory.
Iterative Regularization for Learning with Convex Loss Functions
We consider the problem of supervised learning with convex loss functions and propose a new form of iterative regularization based on the subgradient method.
Minimax Optimal Rates of Estimation in High Dimensional Additive Models: Universal Phase Transition
We establish minimax optimal rates of convergence for estimation in a high dimensional additive model assuming that it is approximately sparse.
Unregularized Online Learning Algorithms with General Loss Functions
Firstly, we derive explicit convergence rates of the unregularized online learning algorithms for classification associated with a general gamma-activating loss (see Definition 1 in the paper).
Online Pairwise Learning Algorithms with Kernels
In this paper, we study an online algorithm for pairwise learning with a least-square loss function in an unconstrained setting of a reproducing kernel Hilbert space (RKHS), which we refer to as the Online Pairwise lEaRning Algorithm (OPERA).
Consistency Analysis of an Empirical Minimum Error Entropy Algorithm
The error entropy consistency, which requires the error entropy of the learned function to approximate the minimum error entropy, is shown to be always true if the bandwidth parameter tends to 0 at an appropriate rate.
Learning rates for the risk of kernel based quantile regression estimators in additive models
Additive models play an important role in semiparametric statistics.
