Search Results for author:
Found 78 papers, 9 papers with code
High-dimensional logistic regression with missing data: Imputation, regularization, and universality
When both the covariates and the additive corruptions are independent and normally distributed, we provide exact characterizations of both the prediction error as well as the estimation error.
Scaling Training Data with Lossy Image Compression
In order to capture this trade-off and optimize storage of training data, we propose a `storage scaling law' that describes the joint evolution of test error with sample size and number of bits per image.
Which exceptional low-dimensional projections of a Gaussian point cloud can be found in polynomial time?
Given $d$-dimensional standard Gaussian vectors $\boldsymbol{x}_1,\dots, \boldsymbol{x}_n$, we consider the set of all empirical distributions of its $m$-dimensional projections, for $m$ a fixed constant.
Scaling laws for learning with real and surrogate data
We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains.
Universality of max-margin classifiers
Working in the high-dimensional regime in which the number of features $p$, the number of samples $n$ and the input dimension $d$ (in the nonlinear featurization setting) diverge, with ratios of order one, we prove a universality result establishing that the asymptotic behavior is completely determined by the expected covariance of feature vectors and by the covariance between features and labels.
Towards a statistical theory of data selection under weak supervision
Given a sample of size $N$, it is often useful to select a subsample of smaller size $n<N$ to be used for statistical estimation or learning.
Six Lectures on Linearized Neural Networks
In these six lectures, we examine what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models.
Sampling, Diffusions, and Stochastic Localization
Diffusions are a successful technique to sample from high-dimensional distributions can be either explicitly given or learnt from a collection of samples.
Learning time-scales in two-layers neural networks
In this paper, we study the gradient flow dynamics of a wide two-layer neural network in high-dimension, when data are distributed according to a single-index model (i. e., the target function depends on a one-dimensional projection of the covariates).
Dimension free ridge regression
However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix.
Overparametrized linear dimensionality reductions: From projection pursuit to two-layer neural networks
Denoting by $\mathscr{F}_{m, \alpha}$ the set of probability distributions in $\mathbb{R}^m$ that arise as low-dimensional projections in this limit, we establish new inner and outer bounds on $\mathscr{F}_{m, \alpha}$.
Adversarial Examples in Random Neural Networks with General Activations
A substantial body of empirical work documents the lack of robustness in deep learning models to adversarial examples.
Universality of empirical risk minimization
In particular, the asymptotics of these quantities can be computed $-$to leading order$-$ under a simpler model in which the feature vectors ${\boldsymbol x}_i$ are replaced by Gaussian vectors ${\boldsymbol g}_i$ with the same covariance.
Tractability from overparametrization: The example of the negative perceptron
In the negative perceptron problem we are given $n$ data points $({\boldsymbol x}_i, y_i)$, where ${\boldsymbol x}_i$ is a $d$-dimensional vector and $y_i\in\{+1,-1\}$ is a binary label.
Streaming Belief Propagation for Community Detection
The community detection problem requires to cluster the nodes of a network into a small number of well-connected "communities".
Minimum complexity interpolation in random features models
We study random features approximations to these norms and show that, for $p>1$, the number of random features required to approximate the original learning problem is upper bounded by a polynomial in the sample size.
Deep learning: a statistical viewpoint
We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting.
Learning with invariances in random features and kernel models
Certain neural network architectures -- for instance, convolutional networks -- are believed to owe their success to the fact that they exploit such invariance properties.
Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration
We show that the test error of random features ridge regression is dominated by its approximation error and is larger than the error of KRR as long as $N\le n^{1-\delta}$ for some $\delta>0$.
Underspecification Presents Challenges for Credibility in Modern Machine Learning
Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains.
The Lasso with general Gaussian designs with applications to hypothesis testing
On the other hand, the Lasso estimator can be precisely characterized in the regime in which both $n$ and $p$ are large and $n/p$ is of order one.
The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training
We assume that both the sample size $n$ and the dimension $d$ are large, and they are polynomially related.
When Do Neural Networks Outperform Kernel Methods?
Recent empirical work showed that, for some classification tasks, RKHS methods can replace NNs without a large loss in performance.
The estimation error of general first order methods
These lower bounds are optimal in the sense that there exist algorithms whose estimation error matches the lower bounds up to asymptotically negligible terms.
Imputation for High-Dimensional Linear Regression
We study high-dimensional regression with missing entries in the covariates.
Limitations of Lazy Training of Two-layers Neural Network
We study the supervised learning problem under either of the following two models: (1) Feature vectors x_i are d-dimensional Gaussian and responses are y_i = f_*(x_i) for f_* an unknown quadratic function; (2) Feature vectors x_i are distributed as a mixture of two d-dimensional centered Gaussians, and y_i's are the corresponding class labels.
The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
They achieve this by learning nonlinear representations of the inputs that maps the data into linearly separable classes.
The generalization error of random features regression: Precise asymptotics and double descent curve
We compute the precise asymptotics of the test error, in the limit $N, n, d\to \infty$ with $N/d$ and $n/d$ fixed.
Limitations of Lazy Training of Two-layers Neural Networks
We study the supervised learning problem under either of the following two models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic function; (2) Feature vectors ${\boldsymbol x}_i$ are distributed as a mixture of two $d$-dimensional centered Gaussians, and $y_i$'s are the corresponding class labels.
Linearized two-layers neural networks in high dimension
Both these approaches can also be regarded as randomized approximations of kernel ridge regression (with respect to different kernels), and enjoy universal approximation properties when the number of neurons $N$ diverges, for a fixed dimension $d$.
Surprises in High-Dimensional Ridgeless Least Squares Interpolation
Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type.
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
Earlier work shows that (under some regularity assumptions), the mean field description is accurate as soon as the number of hidden units is much larger than the dimension $D$.
Analysis of a Two-Layer Neural Network via Displacement Convexity
We prove that, in the limit in which the number of neurons diverges, the evolution of gradient descent converges to a Wasserstein gradient flow in the space of probability distributions over $\Omega$.
Contextual Stochastic Block Models
We provide the first information theoretic tight analysis for inference of latent community structure given a sparse graph along with high dimensional node covariates, correlated with the same latent communities.
A Mean Field View of the Landscape of Two-Layers Neural Networks
Does SGD converge to a global optimum of the risk or only to a local optimum?
On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
Our conclusion holds for a `natural data distribution', namely standard Gaussian feature vectors $\boldsymbol x$, and output distributed according to a two-layer neural network with random isotropic weights, and under a certain complexity-theoretic assumption on tensor decomposition.
An Instability in Variational Inference for Topic Models
Namely, for certain regimes of the model parameters, variational inference outputs a non-trivial decomposition into topics.
The landscape of the spiked tensor model
We compute the expected number of critical points and local maxima of this objective function and show that it is exponential in the dimensions $n$, and give exact formulas for the exponential growth rate.
Estimation of Low-Rank Matrices via Approximate Message Passing
In this paper we present a practical algorithm that can achieve Bayes-optimal accuracy above the spectral threshold.
Inference in Graphical Models via Semidefinite Programming Hierarchies
We demonstrate that the resulting algorithm can solve problems with tens of thousands of variables within minutes, and outperforms BP and GBP on practical problems such as image denoising and Ising spin glasses.
Learning Combinations of Sigmoids Through Gradient Estimation
In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients.
Fundamental Limits of Weak Recovery with Applications to Phase Retrieval
In phase retrieval we want to recover an unknown signal $\boldsymbol x\in\mathbb C^d$ from $n$ quadratic measurements of the form $y_i = |\langle{\boldsymbol a}_i,{\boldsymbol x}\rangle|^2+w_i$ where $\boldsymbol a_i\in \mathbb C^d$ are known sensing vectors and $w_i$ is measurement noise.
Non-negative Matrix Factorization via Archetypal Analysis
In this paper, we study an approach to NMF that can be traced back to the work of Cutler and Breiman (1994) and does not require the data to be separable, while providing a generally unique decomposition.
Solving SDPs for synchronization and MaxCut problems via the Grothendieck inequality
In this paper we study the rank-constrained version of SDPs arising in MaxCut and in synchronization problems.
Spectral algorithms for tensor completion
In the tensor completion problem, one seeks to estimate a low-rank tensor based on a random sample of revealed entries.
How Well Do Local Algorithms Solve Semidefinite Programs?
Several probabilistic models from high-dimensional statistics and machine learning reveal an intriguing --and yet poorly understood-- dichotomy.
The Landscape of Empirical Risk for Non-convex Losses
We establish uniform convergence of the gradient and Hessian of the empirical risk to their population counterparts, as soon as the number of samples becomes larger than the number of unknown parameters (modulo logarithmic factors).
Performance of a community detection algorithm based on semidefinite programming
In this paper we study in detail several practical aspects of this new algorithm based on semidefinite programming for the detection of the planted partition.
Online Rules for Control of False Discovery Rate and False Discovery Exceedance
In this paper we consider the problem of controlling FDR in an "online manner".
A Grothendieck-type inequality for local maxima
A large number of problems in optimization, machine learning, signal processing can be effectively addressed by suitable semidefinite programming (SDP) relaxations.
On the Limitation of Spectral Methods: From the Gaussian Hidden Clique Problem to Rank-One Perturbations of Gaussian Tensors
We consider the following detection problem: given a realization of asymmetric matrix $X$ of dimension $n$, distinguish between the hypothesisthat all upper triangular variables are i. i. d.
Convergence rates of sub-sampled Newton methods
In this regime, algorithms which utilize sub-sampling techniques are known to be effective.
De-biasing the Lasso: Optimal Sample Size for Gaussian Designs
When the covariance is known, we prove that the debiased estimator is asymptotically Gaussian under the nearly optimal condition $s_0 = o(n/ (\log p)^2)$.
Improved Sum-of-Squares Lower Bounds for Hidden Clique and Hidden Submatrix Problems
Here we consider the degree-$4$ SOS relaxation, and study the construction of \cite{meka2013association} to prove that SOS fails unless $k\ge C\, n^{1/3}/\log n$.
On Online Control of False Discovery Rate
Given a sequence of null hypotheses $\mathcal{H}(n) = (H_1,..., H_n)$, Benjamini and Hochberg \cite{benjamini1995controlling} introduced the false discovery rate (FDR) criterion, which is the expected proportion of false positives among rejected null hypotheses, and proposed a testing procedure that controls FDR below a pre-assigned significance level.
Finding One Community in a Sparse Graph
This can be regarded as a model for the problem of finding a tightly knitted community in a social network, or a cluster in a relational dataset.
Cone-Constrained Principal Component Analysis
We consider a simple model for noisy quadratic observation of an unknown vector $\bvz$.
A statistical model for tensor PCA
This is possibly related to a fundamental limitation of computationally tractable estimators for this problem.
Statistical Estimation: From Denoising to Sparse Regression and Hidden Cliques
These notes review six lectures given by Prof. Andrea Montanari on the topic of statistical estimation for linear models.
Computational Implications of Reducing Data to Sufficient Statistics
Given a large dataset and an estimation task, it is common to pre-process the data by reducing them to a set of sufficient statistics.
Guess Who Rated This Movie: Identifying Users Through Subspace Clustering
It is often the case that, within an online recommender system, multiple users share a common account.
Privacy Tradeoffs in Predictive Analytics
Recent research has demonstrated that several private user attributes (such as political affiliation, sexual orientation, and gender) can be inferred from such data.
Confidence Intervals and Hypothesis Testing for High-Dimensional Statistical Models
This in turn implies that it is extremely challenging to quantify the `uncertainty' associated with a certain parameter estimate.
Estimating LASSO Risk and Noise Level
In this context, we develop new estimators for the $\ell_2$ estimation risk $\|\hat{\theta}-\theta_0\|_2$ and the variance of the noise.
Sparse PCA via Covariance Thresholding
In an influential paper, \cite{johnstone2004sparse} introduced a simple algorithm that estimates the support of the principal vectors $\mathbf{v}_1,\dots,\mathbf{v}_r$ by the largest entries in the diagonal of the empirical covariance.
Learning Mixtures of Linear Classifiers
We consider a discriminative learning (regression) problem, whereby the regression function is a convex combination of k linear classifiers.
Nearly Optimal Sample Size in Hypothesis Testing for High-Dimensional Regression
In the regime where the number of parameters $p$ is comparable to or exceeds the sample size $n$, a successful approach uses an $\ell_1$-penalized least squares estimator, known as Lasso.
Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
This in turn implies that it is extremely challenging to quantify the \emph{uncertainty} associated with a certain parameter estimate.
Model Selection for High-Dimensional Regression under the Generalized Irrepresentability Condition
In the high-dimensional regression model a response variable is linearly related to $p$ covariates, but the sample size $n$ is smaller than $p$.
Hypothesis Testing in High-Dimensional Regression under the Gaussian Random Design Model: Asymptotic Theory
In this case we prove that a similar distributional characterization (termed `standard distributional limit') holds for $n$ much larger than $s_0(\log p)^2$.
Accelerated Time-of-Flight Mass Spectrometry
We study a simple modification to the conventional time of flight mass spectrometry (TOFMS) where a \emph{variable} and (pseudo)-\emph{random} pulsing rate is used which allows for traces from different pulses to overlap.
Learning Networks of Stochastic Differential Equations
We consider linear models for stochastic dynamics.
The LASSO risk: asymptotic results and real world examples
We consider the problem of learning a coefficient vector x0 from noisy linear observation y=Ax0+w.
The Noise-Sensitivity Phase Transition in Compressed Sensing
We develop formal expressions for the MSE of \hxl, and evaluate its worst-case formal noise sensitivity over all types of k-sparse signals.
Statistics Theory Information Theory Information Theory Statistics Theory
Which graphical models are difficult to learn?
We consider the problem of learning the structure of Ising models (pairwise binary Markov random fields) from i. i. d.
Matrix Completion from Noisy Entries
Given a matrix M of low-rank, we consider the problem of reconstructing it from noisy observations of a small, random subset of its entries.
Matrix Completion from a Few Entries
In the process of proving these statements, we obtain a generalization of a celebrated result by Friedman-Kahn-Szemeredi and Feige-Ofek on the spectrum of sparse random matrices.
Solving Constraint Satisfaction Problems through Belief Propagation-guided decimation
Message passing algorithms have proved surprisingly successful in solving hard constraint satisfaction problems on sparse random graphs.
