Search Results for author:
Found 27 papers, 4 papers with code
Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime
On the other hand, if $p = o(n)$, the number of random features $p$ is the limiting factor and RFRR test error matches the approximation error of the random feature model class (akin to taking $n = \infty$).
A Convergence Analysis of Approximate Message Passing with Non-Separable Functions and Applications to Multi-Class Classification
Motivated by the recent application of approximate message passing (AMP) to the analysis of convex optimizations in multi-class classifications [Loureiro, et.
Asymptotics of feature learning in two-layer networks after one gradient-step
To our knowledge, our results provides the first tight description of the impact of feature learning in the generalization of two-layer neural networks in the large learning rate regime $\eta=\Theta_{d}(d)$, beyond perturbative finite width corrections of the conjugate and neural tangent kernels.
Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime
Earlier work by various authors showed that, when the columns of $X$ are either uniform on the sphere or standard Gaussian vectors, and when $\ell$ is an integer (the linear regime $\ell = 1$ is particularly well-studied), the bulk eigenvalues of such matrices behave in a simple way: They are asymptotically given by the free convolution of the semicircular and Mar\v{c}enko-Pastur distributions, with relative weights given by expanding $f$ in the Hermite basis.
Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improvements under different model and data scaling regimes.
Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime
The generalization performance of kernel ridge regression (KRR) exhibits a multi-phased pattern that crucially depends on the scaling relationship between the sample size $n$ and the underlying dimension $d$.
An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalings
Our work reveals an equivalence principle: the spectrum of the random kernel matrix is asymptotically equivalent to that of a simpler matrix model, constructed as a linear combination of a (shifted) Wishart matrix and an independent matrix sampled from the Gaussian orthogonal ensemble.
Analysis of Random Sequential Message Passing Algorithms for Approximate Inference
We analyze the dynamics of a random sequential message passing algorithm for approximate inference with large Gaussian latent variable models in a student-teacher scenario.
On the Inherent Regularization Effects of Noise Injection During Training
Randomly perturbing networks during the training process is a commonly used approach to improving generalization performance.
Householder Dice: A Matrix-Free Algorithm for Simulating Dynamics on Gaussian and Random Orthogonal Ensembles
This paper proposes a new algorithm, named Householder Dice (HD), for simulating dynamics on dense random matrix ensembles with translation-invariant properties.
Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task.
Construction of optimal spectral methods in phase retrieval
We consider the phase retrieval problem, in which the observer wishes to recover a $n$-dimensional real or complex signal $\mathbf{X}^\star$ from the (possibly noisy) observation of $|\mathbf{\Phi} \mathbf{X}^\star|$, in which $\mathbf{\Phi}$ is a matrix of size $m \times n$.
Information Theory Disordered Systems and Neural Networks Information Theory
Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization
We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer neural network with random iid inputs.
The role of regularization in classification of high-dimensional noisy Gaussian mixture
We also illustrate the interpolation peak at low regularization, and analyze the role of the respective sizes of the two clusters.
Generalized Approximate Survey Propagation for High-Dimensional Estimation
In Generalized Linear Estimation (GLE) problems, we seek to estimate a signal that is observed through a linear transform followed by a component-wise, possibly nonlinear and noisy, channel.
SLOPE for Sparse Linear Regression:Asymptotics and Optimal Regularization
In sparse linear regression, the SLOPE estimator generalizes LASSO by penalizing different coordinates of the estimate according to their magnitudes.
Information Theory Information Theory Statistics Theory Statistics Theory
Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
Substantial progress has been made recently on developing provably accurate and efficient algorithms for low-rank matrix factorization via nonconvex optimization.
MMSE Approximation For Sparse Coding Algorithms Using Stochastic Resonance
The proposed method adds controlled noise to the input and estimates a sparse representation from the perturbed signal.
Streaming PCA and Subspace Tracking: The Missing Data Case
This survey article reviews a variety of classical and recent algorithms for solving this problem with low computational and memory complexities, particularly those applicable in the big data regime with missing data.
A Solvable High-Dimensional Model of GAN
We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data.
Subspace Estimation from Incomplete Observations: A High-Dimensional Analysis
In addition to providing asymptotically exact predictions of the dynamic performance of the algorithms, our high-dimensional analysis yields several insights, including an asymptotic equivalence between Oja's method and GROUSE, and a precise scaling relationship linking the amount of missing data to the signal-to-noise ratio.
Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA
In addition to characterizing the dynamic performance of online learning algorithms, our asymptotic analysis also provides useful insights.
The Scaling Limit of High-Dimensional Online Independent Component Analysis
As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measure of the target feature vector and the estimates provided by the algorithm will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE.
Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimation
We study a spectral initialization method that serves a key role in recent work on estimating signals in nonconvex settings.
A Tale of Two Bases: Local-Nonlocal Regularization on Image Patches with Convolution Framelets
We propose an image representation scheme combining the local and nonlocal characterization of patches in an image.
Understanding Symmetric Smoothing Filters: A Gaussian Mixture Model Perspective
We show that Sinkhorn-Knopp is equivalent to an Expectation-Maximization (EM) algorithm of learning a Gaussian mixture model of the image patches.
Monte Carlo non local means: Random sampling for large-scale image filtering
In particular, our error probability bounds show that, at any given sampling ratio, the probability for MCNLM to have a large deviation from the original NLM solution decays exponentially as the size of the image or database grows.
