Search Results for author:
Found 33 papers, 21 papers with code
Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs
We study the impact of the batch size $n_b$ on the iteration time $T$ of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target functions of isotropic covariates.
Dimension-free deterministic equivalents for random feature regression
Our main contribution is a general deterministic equivalent for the test error of RFRR.
Fundamental computational limits of weak learnability in high-dimensional multi-index models
Multi-index models - functions which only depend on the covariates through a non-linear transformation of their projection on a subspace - are a useful benchmark for investigating feature learning with neural networks.
Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression
We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dimensional supervised regression tasks.
Asymptotics of Learning with Deep Structured (Random) Features
For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of training samples are proportionally large.
A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs
This work investigates adversarial training in the context of margin-based linear classifiers in the high-dimensional regime where the dimension $d$ and the number of data points $n$ diverge with a fixed ratio $\alpha = n / d$.
Asymptotics of feature learning in two-layer networks after one gradient-step
In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step.
High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality
In particular, we provide a sharp asymptotic characterisation of M-estimators trained on a family of elliptical covariate and noise data distributions including cases where second and higher moments do not exist.
Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD
These insights are grounded in the reduction of SGD dynamics to a stochastic process in lower dimensions, where escaping mediocrity equates to calculating an exit time.
How Two-Layer Neural Networks Learn, One (Giant) Step at a Time
The picture drastically improves over multiple gradient steps: we show that a batch-size of $n = \mathcal{O}(d)$ is indeed enough to learn multiple target directions satisfying a staircase property, where more and more directions can be learned over time.
Expectation consistency for calibration of neural networks
Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence.
Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation
Motivated by the recent stream of results on the Gaussian universality of the test and training errors in generalized linear estimation, we ask ourselves the question: "when is a single Gaussian enough to characterize the error?".
From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks
This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a two-layer neural network trained on Gaussian data and labels generated by a similar, though not necessarily identical, target function.
Deterministic equivalent and error universality of deep random features learning
Establishing this result requires proving a deterministic equivalent for traces of the deep random features sample covariance matrices which can be of independent interest.
On double-descent in uncertainty quantification in overparametrized models
Uncertainty quantification is a central challenge in reliable and trustworthy machine learning.
Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
A simple model to study subspace clustering is the high-dimensional $k$-Gaussian mixture model where the cluster means are sparse vectors.
Gaussian Universality of Perceptrons with Random Labels
We argue that there is a large universality class of high-dimensional input data for which we obtain the same minimum training loss as for Gaussian data with corresponding data covariance.
Learning curves for the multi-class teacher-student perceptron
For Gaussian teacher weights, we investigate the performance of ERM with both cross-entropy and square losses, and explore the role of ridge regularisation in approaching Bayes-optimality.
Theoretical characterization of uncertainty in high-dimensional linear classification
In this manuscript, we characterise uncertainty for learning from limited number of samples of high-dimensional Gaussian input data and labels generated by the probit model.
Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks
Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent.
Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension
From the sampling of data to the initialisation of parameters, randomness is ubiquitous in modern Machine Learning practice.
Error Scaling Laws for Kernel Classification under Source and Capacity Conditions
We find that our rates tightly describe the learning curves for this class of data sets, and are also observed on real data.
Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning
Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature of statistical learning.
Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions
Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks.
Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks.
Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization.
Learning curves of generic features maps for realistic datasets with a teacher-student model
While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework.
The Gaussian equivalence of generative models for learning with shallow neural networks
Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models.
Phase retrieval in high dimensions: Statistical and computational phase transitions
We consider the phase retrieval problem of reconstructing a $n$-dimensional real or complex signal $\mathbf{X}^{\star}$ from $m$ (possibly noisy) observations $Y_\mu = | \sum_{i=1}^n \Phi_{\mu i} X^{\star}_i/\sqrt{n}|$, for a large class of correlated real and complex random sensing matrices $\mathbf{\Phi}$, in a high-dimensional setting where $m, n\to\infty$ while $\alpha = m/n=\Theta(1)$.
Generalisation error in learning with random features and the hidden manifold model
In particular, we show how to obtain analytically the so-called double descent behaviour for logistic regression with a peak at the interpolation threshold, we illustrate the superiority of orthogonal against random Gaussian projections in learning with random features, and discuss the role played by correlations in the data generated by the hidden manifold model.
Exact asymptotics for phase retrieval and compressed sensing with random generative priors
We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix.
Precise asymptotics for phase retrieval and compressed sensing with random generative priors
We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix.
The spiked matrix model with generative priors
Here, we replace the sparsity assumption by generative modelling, and investigate the consequences on statistical and algorithmic properties.
