Search Results for author:
Found 96 papers, 8 papers with code
Mechanistic Design and Scaling of Hybrid Architectures
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation.
Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective
In this paper, we study the feature learning ability of two-layer neural networks in the mean-field regime through the lens of kernel methods.
How do Transformers perform In-Context Autoregressive Learning?
More precisely, focusing on commuting orthogonal matrices $W$, we first show that a trained one-layer linear Transformer implements one step of gradient descent for the minimization of an inner objective function, when considering augmented tokens.
Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape
However, existing theoretical studies on how this phenomenon arises are limited to the dynamics of a single layer of attention trained on linear regression tasks.
Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems
In this paper, we extend mean-field Langevin dynamics to minimax optimization over probability distributions for the first time with symmetric and provably convergent updates.
Scalable Federated Learning for Clients with Different Input Image Sizes and Numbers of Output Categories
In this paper, we propose an effective federated learning method named ScalableFL, where the depths and widths of the local models for each client are adjusted according to the clients' input image size and the numbers of output categories.
Learning Green's Function Efficiently Using Low-Rank Approximations
Learning the Green's function using deep learning models enables to solve different classes of partial differential equations.
Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective
Graph neural networks (GNNs) have pioneered advancements in graph representation learning, exhibiting superior feature learning and performance over multilayer perceptrons (MLPs) when handling graph inputs.
Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction
Despite the generality of our results, we achieve an improved convergence rate in both the SGD and SVRG settings when specialized to the standard Langevin dynamics.
Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input
Despite the great success of Transformer networks in various applications such as natural language processing and computer vision, their theoretical aspects are not well understood.
Tight and fast generalization error bound of graph embedding in metric space
However, recent theoretical analyses have shown a much higher upper bound on non-Euclidean graph embedding's generalization error than Euclidean one's, where a high generalization error indicates that the incompleteness and noise in the data can significantly damage learning performance.
Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems
The entropic fictitious play (EFP) is a recently proposed algorithm that minimizes the sum of a convex functional and entropy in the space of measures -- such an objective naturally arises in the optimization of a two-layer neural network in the mean-field regime.
Diffusion Models are Minimax Optimal Distribution Estimators
While efficient distribution learning is no doubt behind the groundbreaking success of diffusion modeling, its theoretical guarantees are quite limited.
Koopman-based generalization bound: New aspect for full-rank weights
Our bound is tighter than existing norm-based bounds when the condition numbers of weight matrices are small.
DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning
In the previous work, the best known utility bound is $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ in terms of the squared full gradient norm, which is achieved by Differential Private Gradient Descent (DP-GD) as an instance, where $n$ is the sample size, $d$ is the problem dimensionality and $\varepsilon_\mathrm{DP}$ is the differential privacy parameter.
Graph Polynomial Convolution Models for Node Classification of Non-Homophilous Graphs
Additionally, we propose adaptive learning between directly graph polynomial convolution models and learning directly from the adjacency matrix.
Versatile Single-Loop Method for Gradient Estimator: First and Second Order Optimality, and its Application to Federated Learning
While variance reduction methods have shown great success in solving large scale optimization problems, many of them suffer from accumulated errors and, therefore, should periodically require the full gradient computation.
Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods
While deep learning has outperformed other methods for various tasks, theoretical frameworks that explain its reason have not been fully established.
High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation
We study the first gradient descent step on the first-layer parameters $\boldsymbol{W}$ in a two-layer neural network: $f(\boldsymbol{x}) = \frac{1}{\sqrt{N}}\boldsymbol{a}^\top\sigma(\boldsymbol{W}^\top\boldsymbol{x})$, where $\boldsymbol{W}\in\mathbb{R}^{d\times N}, \boldsymbol{a}\in\mathbb{R}^{N}$ are randomly initialized, and the training objective is the empirical MSE loss: $\frac{1}{n}\sum_{i=1}^n (f(\boldsymbol{x}_i)-y_i)^2$.
Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimization
The stochastic gradient Langevin Dynamics is one of the most fundamental algorithms to solve sampling problems and non-convex optimization appearing in several machine learning applications.
Convergence Error Analysis of Reflected Gradient Langevin Dynamics for Globally Optimizing Non-Convex Constrained Problems
Gradient Langevin dynamics and a variety of its variants have attracted increasing attention owing to their convergence towards the global optimal solution, initially in the unconstrained convex framework while recently even in convex constrained non-convex problems.
Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning
In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.
Convex Analysis of the Mean Field Langevin Dynamics
In this work, we give a concise and self-contained convergence rate analysis of the mean field Langevin dynamics with respect to the (regularized) objective function in both continuous and discrete time settings.
A Scaling Law for Syn-to-Real Transfer: How Much Is Your Pre-training Effective?
Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks.
Learnability of convolutional neural networks for infinite dimensional input via mixed and anisotropic smoothness
Although the approximation and estimation errors of neural networks are affected by the curse of dimensionality in the existing analyses for typical function spaces such as the \Holder and Besov spaces, we show that, by considering anisotropic smoothness, they can alleviate exponential dependency on the dimensionality but they only depend on the smoothness of the target functions.
Understanding the Variance Collapse of SVGD in High Dimensions
Stein variational gradient descent (SVGD) is a deterministic inference algorithm that evolves a set of particles to fit a target distribution.
Particle Stochastic Dual Coordinate Ascent: Exponential convergent algorithm for mean field neural network optimization
We introduce Particle-SDCA, a gradient-based optimization algorithm for two-layer neural networks in the mean field regime that achieves exponential convergence rate in regularized empirical risk minimization.
Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime
Generalization measures are intensively studied in the machine learning community for better modeling generalization gaps.
A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks.
Layer-wise Adaptive Graph Convolution Networks Using Generalized Pagerank
We investigate adaptive layer-wise graph convolution in deep GCN models.
AutoLL: Automatic Linear Layout of Graphs based on Deep Neural Network
However, it is limited to a two-mode reordering (i. e., the rows and columns are reordered separately) and it cannot be applied in the one-mode setting (i. e., the same node order is used for reordering both rows and columns), owing to the characteristics of its model architecture.
On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting
Deep learning empirically achieves high performance in many applications, but its training dynamics has not been fully understood theoretically.
Particle Dual Averaging: Optimization of Mean Field Neural Network with Global Convergence Rate Analysis
An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain.
Deep Two-Way Matrix Reordering for Relational Data Analysis
This denoised mean matrix can be used to visualize the global structure of the reordered observed matrix.
A Goodness-of-fit Test on the Number of Biclusters in a Relational Data Matrix
Biclustering is a method for detecting homogeneous submatrices in a given observed matrix, and it is an effective tool for relational data analysis.
Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning
Recently, local SGD has got much attention and been extensively studied in the distributed learning community to overcome the communication bottleneck problem.
Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain.
Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods
Establishing a theoretical analysis that explains why deep learning can outperform shallow learning such as kernel methods is one of the biggest issues in the deep learning literature.
Estimation error analysis of deep learning on the regression problem on the variable exponent Besov space
In this study, we focus on the adaptivity of deep learning; consequently, we treat the variable exponent Besov space, which has a different smoothness depending on the input location $x$.
MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search
To overcome this problem, we propose a method called minimum stable rank DARTS (MSR-DARTS), for finding a model with the best generalization error by replacing architecture optimization with the selection process using the minimum stable rank criterion.
Ranked #24 on
Neural Architecture Search
on CIFAR-10
Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding
According to the Rate-distortion theory, the optimal transform coding is achieved by using an orthonormal transform with PCA basis where the transform space is isometric to the input.
Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics
Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
In this study, we show that the averaged stochastic gradient descent can achieve the minimax optimal convergence rate, with the global convergence guarantee, by exploiting the complexities of the target function and the RKHS associated with the NTK.
Gradient Descent in RKHS with Importance Labeling
In this paper, we study importance labeling problem, in which we are given many unlabeled data and select a limited number of data to be labeled from the unlabeled data, and then a learning algorithm is executed on the selected one.
When Does Preconditioning Help or Hurt Generalization?
While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question.
Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks
By combining it with generalization gap bounds in terms of transductive Rademacher complexity, we show that a test error bound of a specific type of multi-scale GNNs that decreases corresponding to the number of node aggregations under some conditions.
Selective Inference for Latent Block Models
In this case, it becomes crucial to consider the selective bias in the block structure, that is, the block structure is selected from all the possible cluster memberships based on some criterion by the clustering algorithm.
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint
This paper investigates the generalization properties of two-layer neural networks in high-dimensions, i. e. when the number of samples $n$, features $d$, and neurons $h$ tend to infinity at the same rate.
Meta Cyclical Annealing Schedule: A Simple Approach to Avoiding Meta-Amortization Error
To address this challenge, we design a novel meta-regularization objective using {\it cyclical annealing schedule} and {\it maximum mean discrepancy} (MMD) criterion.
Dimension-free convergence rates for gradient Langevin dynamics in RKHS
In this work, we provide a convergence analysis of GLD and SGLD when the optimization space is an infinite dimensional Hilbert space.
Understanding Generalization in Deep Learning via Tensor Methods
Recently proposed complexity measures have provided insights to understanding the generalizability in neural networks from perspectives of PAC-Bayes, robustness, overparametrization, compression and so on.
Domain Adaptation Regularization for Spectral Pruning
We also show that our method outperforms an existing compression method studied in the DA setting by a large margin for high compression rates.
Exponential Convergence Rates of Classification Errors on Learning with SGD and Random Features
To address this problem, sketching and stochastic gradient methods are the most commonly used techniques to derive efficient large-scale learning algorithms.
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks
Compressing DNNs is important for the real-world applications operating on resource-constrained devices.
Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space
The results show that deep learning has better dependence on the input dimensionality if the target function possesses anisotropic smoothness, and it achieves an adaptive rate for functions with spatially inhomogeneous smoothness.
Towards Characterizing the High-dimensional Bias of Kernel-based Particle Inference Algorithms
Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles.
Scalable Deep Neural Networks via Low-Rank Matrix Factorization
Compressing deep neural networks (DNNs) is important for real-world applications operating on resource-constrained devices.
Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
However, the compression based bound can be applied only to a compressed network, and it is not applicable to the non-compressed original network.
Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum
Furthermore, we propose a method for automatically determining the widths (the numbers of channels) of object detectors based on the eigenspectrum.
Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep learning because of its training-time efficiency, however, extremely large-batch SGD leads to poor generalization and easily converges to sharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from converging to good minima.
Goodness-of-fit Test for Latent Block Models
Latent block models are used for probabilistic biclustering, which is shown to be an effective method for analyzing various relational data sets.
Accelerated Sparsified SGD with Error Feedback
Several work has shown that {\it{sparsified}} stochastic gradient descent method (SGD) with {\it{error feedback}} asymptotically achieves the same rate as (non-sparsified) parallel SGD.
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
We show that when the Erd\H{o}s -- R\'{e}nyi graph is sufficiently dense and large, a broad range of GCNs on it suffers from the "information loss" in the limit of infinite layers with high probability.
Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
Most studies especially focused on the regression problems with the squared loss function, except for a few, and the importance of the positivity of the neural tangent kernel has been pointed out.
On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
Whereas existing theoretical studies of deep learning have been based mainly on mathematical theories of well-known function classes such as H\"{o}lder and Besov classes, we focus on function classes with discontinuity and sparsity, which are those naturally assumed in practice.
Approximation and non-parametric estimation of ResNet-type convolutional neural networks via block-sparse fully-connected neural networks
We develop new approximation and statistical learning theories of convolutional neural networks (CNNs) via the ResNet-type structure where the channel size, filter size, and width are fixed.
Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks
The key idea is that we can replicate the learning ability of Fully-connected neural networks (FNNs) by tailored CNNs, as long as the FNNs have \textit{block-sparse} structures.
Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation.
Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space.
Sample Efficient Stochastic Gradient Iterative Hard Thresholding Method for Stochastic Sparse Linear Regression with Limited Attribute Observation
We develop new stochastic gradient methods for efficiently solving sparse linear regression in a partial attribute observation setting, where learners are only allowed to observe a fixed number of actively chosen attributes per example at training and prediction times.
Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error
The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based error bound.
Stochastic Gradient Descent with Exponential Convergence Rates of Expected Classification Errors
In this paper, we show an exponential convergence of the expected classification error in the final phase of the stochastic gradient descent for a wide class of differentiable convex loss functions under similar assumptions.
Cross-domain Recommendation via Deep Domain Adaptation
The behavior of users in certain services could be a clue that can be used to infer their preferences and may be used to make recommendations for other services they have never used.
Functional Gradient Boosting based on Residual Network Perception
Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well.
Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models
In this paper, this phenomenon is explained from the functional gradient method perspective of the gradient layer.
Stochastic Particle Gradient Descent for Infinite Ensembles
The superior performance of ensemble methods with infinite models are well known.
Independently Interpretable Lasso: A New Regularizer for Sparse Regression with Uncorrelated Variables
However, one of the biggest issues in sparse regularization is that its performance is quite sensitive to correlations between features.
Fast learning rate of deep learning via a kernel perspective
Our point of view is to deal with the ordinary finite dimensional deep neural network as a finite approximation of the infinite dimensional one.
Trimmed Density Ratio Estimation
Density ratio estimation is a vital tool in both machine learning and statistical community.
Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method for Regularized Empirical Risk Minimization
In this paper, we develop a new accelerated stochastic gradient method for efficiently solving the convex regularized empirical risk minimization problem in mini-batch settings.
Learning Sparse Structural Changes in High-dimensional Markov Networks: A Review on Methodologies and Theories
Recent years have seen an increasing popularity of learning the sparse \emph{changes} in Markov Networks.
Minimax Optimal Alternating Minimization for Kernel Nonparametric Tensor Learning
We investigate the statistical performance and computational efficiency of the alternating minimization procedure for nonparametric tensor learning.
Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning.
Structure Learning of Partitioned Markov Networks
We learn the structure of a Markov Network between two groups of random variables from joint observations.
Convergence rate of Bayesian tensor estimator: Optimal rate without restricted strong convexity
In this paper, we investigate the statistical convergence rate of a Bayesian low-rank tensor estimator.
Spectral norm of random tensors
We show that the spectral norm of a random $n_1\times n_2\times \cdots \times n_K$ tensor (or higher-order array) scales as $O\left(\sqrt{(\sum_{k=1}^{K}n_k)\log(K)}\right)$ under some sub-Gaussian assumption on the entries.
Support Consistency of Direct Sparse-Change Learning in Markov Networks
We study the problem of learning sparse structure changes between two Markov networks $P$ and $Q$.
Stochastic Dual Coordinate Ascent with Alternating Direction Multiplier Method
We propose a new stochastic dual coordinate ascent technique that can be applied to a wide range of regularized learning problems.
Direct Learning of Sparse Changes in Markov Networks by Density Ratio Estimation
We propose a new method for detecting changes in Markov network structure between two sets of samples.
Convex Tensor Decomposition via Structured Schatten Norm Regularization
We discuss structured Schatten norms for tensor decomposition that includes two recently proposed norms ("overlapped" and "latent") for convex-optimization-based tensor decomposition, and connect tensor decomposition with wider literature on structured sparsity.
Density-Difference Estimation
A naive approach is a two-step procedure of first estimating two densities separately and then computing their difference.
Fast learning rate of multiple kernel learning: Trade-off between sparsity and smoothness
If the ground truth is smooth, we show a faster convergence rate for the elastic-net regularization with less conditions than $\ell_1$-regularization; otherwise, a faster convergence rate for the $\ell_1$-regularization is shown.
Unifying Framework for Fast Learning Rate of Non-Sparse Multiple Kernel Learning
Finally, we show that, when the complexities of candidate reproducing kernel Hilbert spaces are inhomogeneous, dense type regularization shows better learning rate compared with sparse ℓ1 regularization.
Relative Density-Ratio Estimation for Robust Distribution Comparison
Divergence estimators based on direct approximation of density-ratios without going through separate approximation of numerator and denominator densities have been successfully applied to machine learning tasks that involve distribution comparison such as outlier detection, transfer learning, and two-sample homogeneity test.
Statistical Performance of Convex Tensor Decomposition
We analyze the statistical performance of a recently proposed convex tensor decomposition algorithm.
Condition Number Analysis of Kernel-based Density Ratio Estimation
We show that the kernel least-squares method has a smaller condition number than a version of kernel mean matching and other M-estimators, implying that the kernel least-squares method has preferable numerical properties.
