Search Results for author:
Found 101 papers, 47 papers with code
Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime
We demonstrate that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization.
Fundamental limits of Non-Linear Low-Rank Matrix Estimation
We consider the task of estimating a low-rank matrix from non-linear and noisy observations.
Spectral Phase Transition and Optimal PCA in Block-Structured Spiked models
We discuss the inhomogeneous spiked Wigner model, a theoretical framework recently introduced to study structured noise in various learning scenarios, through the prism of random matrix theory, with a specific focus on its spectral properties.
Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression
We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dimensional supervised regression tasks.
A High Dimensional Model for Adversarial Training: Geometry and Trade-Offs
This work investigates adversarial training in the context of margin-based linear classifiers in the high-dimensional regime where the dimension $d$ and the number of data points $n$ diverge with a fixed ratio $\alpha = n / d$.
Asymptotics of feature learning in two-layer networks after one gradient-step
To our knowledge, our results provides the first tight description of the impact of feature learning in the generalization of two-layer neural networks in the large learning rate regime $\eta=\Theta_{d}(d)$, beyond perturbative finite width corrections of the conjugate and neural tangent kernels.
A phase transition between positional and semantic learning in a solvable model of dot-product attention
We investigate how a dot-product attention layer learns a positional attention matrix (with tokens attending to each other based on their respective positions) and a semantic attention matrix (with tokens attending to each other based on their meaning).
The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents
In particular, multi-pass GD with finite stepsize is found to overcome the limitations of gradient flow and single-pass GD given by the information exponent (Ben Arous et al., 2021) and leap exponent (Abbe et al., 2023) of the target function.
Analysis of learning a flow-based generative model from limited sample complexity
We study the problem of training a flow-based generative model, parametrized by a two-layer autoencoder, to sample from a high-dimensional Gaussian mixture.
Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
Recent years witnessed the development of powerful generative models based on flows, diffusion or autoregressive neural networks, achieving remarkable success in generating data from examples with applications in a broad range of areas.
Asymptotic Characterisation of Robust Empirical Risk Minimisation Performance in the Presence of Outliers
We study robust linear regression in high-dimension, when both the dimension $d$ and the number of data points $n$ diverge with a fixed ratio $\alpha=n/d$, and study a data model that includes outliers.
Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD
These insights are grounded in the reduction of SGD dynamics to a stochastic process in lower dimensions, where escaping mediocrity equates to calculating an exit time.
How Two-Layer Neural Networks Learn, One (Giant) Step at a Time
The picture drastically improves over multiple gradient steps: we show that a batch-size of $n = \mathcal{O}(d)$ is indeed enough to learn multiple target directions satisfying a staircase property, where more and more directions can be learned over time.
Expectation consistency for calibration of neural networks
Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence.
Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation
Motivated by the recent stream of results on the Gaussian universality of the test and training errors in generalized linear estimation, we ask ourselves the question: "when is a single Gaussian enough to characterize the error?".
From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks
This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a two-layer neural network trained on Gaussian data and labels generated by a similar, though not necessarily identical, target function.
Bayes-optimal Learning of Deep Random Networks of Extensive-width
We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
On double-descent in uncertainty quantification in overparametrized models
Uncertainty quantification is a central challenge in reliable and trustworthy machine learning.
Rigorous dynamical mean field theory for stochastic gradient descent methods
We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e. g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization.
Gaussian Universality of Perceptrons with Random Labels
We argue that there is a large universality class of high-dimensional input data for which we obtain the same minimum training loss as for Gaussian data with corresponding data covariance.
Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
A simple model to study subspace clustering is the high-dimensional $k$-Gaussian mixture model where the cluster means are sparse vectors.
Theoretical characterization of uncertainty in high-dimensional linear classification
In this manuscript, we characterise uncertainty for learning from limited number of samples of high-dimensional Gaussian input data and labels generated by the probit model.
Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks
Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent.
Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension
From the sampling of data to the initialisation of parameters, randomness is ubiquitous in modern Machine Learning practice.
Error Scaling Laws for Kernel Classification under Source and Capacity Conditions
We find that our rates tightly describe the learning curves for this class of data sets, and are also observed on real data.
Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning
Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature of statistical learning.
Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions
Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks.
Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks.
Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization.
Bayesian reconstruction of memories stored in neural networks from their connectivity
The advent of comprehensive synaptic wiring diagrams of large neural circuits has created the field of connectomics and given rise to a number of open research questions.
Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
Here, we show theoretically that two-layer neural networks (2LNN) with only a few hidden neurons can beat the performance of kernel learning on a simple Gaussian mixture classification task.
Learning curves of generic features maps for realistic datasets with a teacher-student model
While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework.
Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
In the white-box setting, our defense works by obfuscating the parameters of the random projection.
Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
We present a photonic accelerator for Direct Feedback Alignment, able to compute random projections with trillions of parameters.
Construction of optimal spectral methods in phase retrieval
We consider the phase retrieval problem, in which the observer wishes to recover a $n$-dimensional real or complex signal $\mathbf{X}^\star$ from the (possibly noisy) observation of $|\mathbf{\Phi} \mathbf{X}^\star|$, in which $\mathbf{\Phi}$ is a matrix of size $m \times n$.
Information Theory Disordered Systems and Neural Networks Information Theory
Epidemic mitigation by statistical inference from contact tracing data
We conclude that probabilistic risk estimation is capable to enhance performance of digital contact tracing and should be considered in the currently developed mobile applications.
The Gaussian equivalence of generative models for learning with shallow neural networks
Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models.
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea.
Reservoir Computing meets Recurrent Kernels and Structured Transforms
Reservoir Computing is a class of simple yet efficient Recurrent Neural Networks where internal weights are fixed at random and only a linear output layer is trained.
Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval
Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem.
Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization
We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer neural network with random iid inputs.
Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)
For sufficiently strongly convex problems, we show that the two-layer vector approximate message passing algorithm (2-MLVAMP) converges, where the convergence analysis is done by checking the stability of an equivalent dynamical system, which gives the result for such problems.
Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow.
Phase retrieval in high dimensions: Statistical and computational phase transitions
We consider the phase retrieval problem of reconstructing a $n$-dimensional real or complex signal $\mathbf{X}^{\star}$ from $m$ (possibly noisy) observations $Y_\mu = | \sum_{i=1}^n \Phi_{\mu i} X^{\star}_i/\sqrt{n}|$, for a large class of correlated real and complex random sensing matrices $\mathbf{\Phi}$, in a high-dimensional setting where $m, n\to\infty$ while $\alpha = m/n=\Theta(1)$.
Light-in-the-loop: using a photonics co-processor for scalable training of neural networks
As neural networks grow larger and more complex and data-hungry, training costs are skyrocketing.
Tree-AMP: Compositional Inference with Tree Approximate Message Passing
We introduce Tree-AMP, standing for Tree Approximate Message Passing, a python package for compositional inference in high-dimensional tree-structured models.
Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant.
The role of regularization in classification of high-dimensional noisy Gaussian mixture
We also illustrate the interpolation peak at low regularization, and analyze the role of the respective sizes of the two clusters.
Generalisation error in learning with random features and the hidden manifold model
In particular, we show how to obtain analytically the so-called double descent behaviour for logistic regression with a peak at the interpolation threshold, we illustrate the superiority of orthogonal against random Gaussian projections in learning with random features, and discuss the role played by correlations in the data generated by the hidden manifold model.
Asymptotic errors for convex penalized linear regression beyond Gaussian matrices
We consider the problem of learning a coefficient vector $x_{0}$ in $R^{N}$ from noisy linear observations $y=Fx_{0}+w$ in $R^{M}$ in the high dimensional limit $M, N$ to infinity with $\alpha=M/N$ fixed.
Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning
Statistical learning theory provides bounds of the generalization gap, using in particular the Vapnik-Chervonenkis dimension and the Rademacher complexity.
Exact asymptotics for phase retrieval and compressed sensing with random generative priors
We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix.
Who is Afraid of Big Bad Minima? Analysis of gradient-flow in spiked matrix-tensor models
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a quantitative theory explaining this behaviour in a spiked matrix-tensor model. Our framework is based on the Kac-Rice analysis of stationary points and a closed-form analysis of gradient-flow originating from statistical physics.
Kernel computations from large-scale random features obtained by Optical Processing Units
Approximating kernel functions with random features (RFs)has been a successful application of random projections for nonparametric estimation.
Modelling the influence of data structure on learning in neural networks: the hidden manifold model
We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a "Gaussian Equivalence Property" (GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times.
Precise asymptotics for phase retrieval and compressed sensing with random generative priors
We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix.
Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Model
Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones.
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data.
Principled Training of Neural Networks with Direct Feedback Alignment
In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles.
On the Universality of Noiseless Linear Estimation with Respect to the Measurement Matrix
In a noiseless linear estimation problem, one aims to reconstruct a vector x* from the knowledge of its linear projections y=Phi x*.
The spiked matrix model with generative priors
Here, we replace the sparsity assumption by generative modelling, and investigate the consequences on statistical and algorithmic properties.
Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models
In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor model.
Generalisation dynamics of online learning in over-parameterised neural networks
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data.
Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference
Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference.
Rank-one matrix estimation: analysis of algorithmic and information theoretic limits by the spatial coupling method
We characterize the detectability phase transitions in a large set of estimation problems, where we show that there exists a gap between what currently known polynomial algorithms (in particular spectral methods and approximate message-passing) can do and what is expected information theoretically.
Spectral Method for Multiplexed Phase Retrieval and Application in Optical Imaging in Complex Media
We introduce a generalized version of phase retrieval called multiplexed phase retrieval.
Approximate message-passing for convex optimization with non-separable penalties
We introduce an iterative optimization scheme for convex objectives consisting of a linear loss and a non-separable penalty, based on the expectation-consistent approximation and the vector approximate message-passing (VAMP) algorithm.
Approximate Survey Propagation for Statistical Inference
Approximate message passing algorithm enjoyed considerable attention in the last decade.
Fundamental limits of detection in the spiked Wigner model
We study the fundamental limits of detecting the presence of an additive rank-one perturbation, or spike, to a Wigner matrix.
The committee machine: Computational to statistical gaps in learning a two-layers neural network
Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks.
Entropy and mutual information in models of deep neural networks
We examine a class of deep learning models with a tractable method to compute information-theoretic quantities.
Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
Non-rigorous predictions for the optimal errors existed for special cases of GLMs, e. g. for the perceptron, in the field of statistical physics based on the so-called replica method.
Streaming Bayesian inference: theoretical limits and mini-batch approximate message-passing
In statistical learning for real-world large-scale data problems, one must often resort to "streaming" algorithms which operate sequentially on small batches of data.
A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
Restricted Boltzmann machines (RBMs) are energy-based neural-networks which are commonly used as the building blocks for deep architectures neural architectures.
Multi-Layer Generalized Linear Estimation
We consider the problem of reconstructing a signal from multi-layered (possibly) non-linear measurements.
Phase transitions and optimal algorithms in high-dimensional Gaussian mixture clustering
We consider the problem of Gaussian mixture clustering in the high-dimensional limit where the data consists of $m$ points in $n$ dimensions, $n, m \rightarrow \infty$ and $\alpha = m/n$ stays finite.
Scaling up Echo-State Networks with multiple light scattering
As a proof of concept, binary networks have been successfully trained to predict the chaotic Mackey-Glass time series.
Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
In this work, we consider compressed sensing reconstruction from $M$ measurements of $K$-sparse structured signals which do not possess a writable correlation model.
Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula
We also show that for a large set of parameters, an iterative algorithm called approximate message-passing is Bayes optimal.
Fast Randomized Semi-Supervised Clustering
We consider the problem of clustering partially labeled data from a minimal number of randomly chosen pairwise comparisons between the items.
Clustering from Sparse Pairwise Measurements
We consider the problem of grouping items into clusters based on few random pairwise comparisons between the items.
Training Restricted Boltzmann Machine via the Thouless-Anderson-Palmer free energy
Restricted Boltzmann machines are undirected neural networks which have been shown tobe effective in many applications, including serving as initializations fortraining deep multi-layer neural networks.
Statistical physics of inference: Thresholds and algorithms
Many questions of fundamental interest in todays science can be formulated as inference problems: Some partial, or noisy, observations are performed over a set of variables and the goal is to recover, or infer, the values of the variables based on the indirect information contained in the measurements.
Random Projections through multiple optical scattering: Approximating kernels at the speed of light
Random projections have proven extremely useful in many signal processing and machine learning applications.
Intensity-only optical compressive imaging using a multiply scattering material and a double phase retrieval approach
In this paper, the problem of compressive imaging is addressed using natural randomization by means of a multiply scattering medium.
MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel
This paper considers probabilistic estimation of a low-rank matrix from non-linear element-wise measurements of its elements.
Matrix Completion from Fewer Entries: Spectral Detectability and Rank Estimation
We propose a spectral algorithm for these two tasks called MaCBetH (for Matrix Completion with the Bethe Hessian).
Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy
Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks.
Phase Transitions in Sparse PCA
We study optimal estimation for sparse principal component analysis when the number of non-zero elements is small but on the same order as the dimension of the data.
Approximate Message Passing with Restricted Boltzmann Machine Priors
Approximate Message Passing (AMP) has been shown to be an excellent statistical approach to signal inference and compressed sensing problem.
Spectral Detection in the Censored Block Model
We describe two spectral algorithms for this task based on the non-backtracking and the Bethe Hessian operators.
Sparse Estimation with the Swept Approximated Message-Passing Algorithm
Approximate Message Passing (AMP) has been shown to be a superior method for inference problems, such as the recovery of signals from sets of noisy, lower-dimensionality measurements, both in terms of reconstruction accuracy and in computational efficiency.
Spectral Clustering of Graphs with the Bethe Hessian
We show that this approach combines the performances of the non-backtracking operator, thus detecting clusters all the way down to the theoretical limit in the stochastic block model, with the computational, theoretical and memory advantages of real symmetric matrices.
Phase transitions and sample complexity in Bayes-optimal matrix factorization
We use the tools of statistical mechanics - the cavity and replica methods - to analyze the achievability and computational tractability of the inference problems in the setting of Bayes-optimal inference, which amounts to assuming that the two matrices have random independent elements generated from some known distribution, and this information is available to the inference algorithm.
Blind Calibration in Compressed Sensing using Message Passing Algorithms
We study numerically the phase diagram of the blind calibration problem, and show that even in cases where convex relaxation is possible, our algorithm requires a smaller number of measurements and/or signals in order to perform well.
Spectral redemption: clustering sparse networks
Spectral algorithms are classic approaches to clustering and community detection in networks.
Model Selection for Degree-corrected Block Models
We present the first principled and tractable approach to model selection between standard and degree-corrected block models, based on new large-graph asymptotics for the distribution of log-likelihood ratios under the stochastic block model, finding substantial departures from classical results for sparse graphs.
Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
We further develop the asymptotic analysis of the corresponding phase diagrams with and without measurement noise, for different distribution of signals, and discuss the best possible reconstruction performances regardless of the algorithm.
Statistical Mechanics Information Theory Information Theory
Statistical physics-based reconstruction in compressed sensing
Compressed sensing is triggering a major evolution in signal acquisition.
Statistical Mechanics Information Theory Information Theory
Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications
In this paper we extend our previous work on the stochastic block model, a commonly used generative model for social and biological networks, and the problem of inferring functional groups or communities from the topology of the network.
Statistical Mechanics Disordered Systems and Neural Networks Social and Information Networks Physics and Society
Phase transition in the detection of modules in sparse networks
We present an asymptotically exact analysis of the problem of detecting communities in sparse random networks.
