Search Results for author:
Found 109 papers, 8 papers with code
Super Non-singular Decompositions of Polynomials and their Application to Robustly Learning Low-degree PTFs
We study the efficient learnability of low-degree polynomial threshold functions (PTFs) in the presence of a constant fraction of adversarial corruptions.
Robust Sparse Estimation for Gaussians with Optimal Error under Huber Contamination
Concretely, for Gaussian robust $k$-sparse mean estimation on $\mathbb{R}^d$ with corruption rate $\epsilon>0$, our algorithm has sample complexity $(k^2/\epsilon^2)\mathrm{polylog}(d/\epsilon)$, runs in sample polynomial time, and approximates the target mean within $\ell_2$-error $O(\epsilon)$.
Robust Second-Order Nonconvex Optimization and Its Application to Low Rank Matrix Sensing
We introduce a general framework for efficiently finding an approximate SOSP with \emph{dimension-independent} accuracy guarantees, using $\widetilde{O}({D^2}/{\epsilon})$ samples where $D$ is the ambient dimension and $\epsilon$ is the fraction of corrupted datapoints.
SQ Lower Bounds for Non-Gaussian Component Analysis with Weaker Assumptions
In particular, we prove near-optimal SQ lower bounds for NGCA under the moment-matching condition only.
Statistical Query Lower Bounds for Learning Truncated Gaussians
We study the problem of estimating the mean of an identity covariance Gaussian in the truncated setting, in the regime when the truncation set comes from a low-complexity family $\mathcal{C}$ of sets.
Robustly Learning Single-Index Models via Alignment Sharpness
We give an efficient learning algorithm, achieving a constant factor approximation to the optimal loss, that succeeds under a range of distributions (including log-concave distributions) and a broad class of monotone and Lipschitz link functions.
How Does Unlabeled Data Provably Help Out-of-Distribution Detection?
Harnessing the power of unlabeled in-the-wild data is non-trivial due to the heterogeneity of both in-distribution (ID) and OOD data.
Agnostically Learning Multi-index Models with Queries
In contrast, algorithms that rely only on random examples inherently require $d^{\mathrm{poly}(1/\epsilon)}$ samples and runtime, even for the basic problem of agnostically learning a single ReLU or a halfspace.
Clustering Mixtures of Bounded Covariance Distributions Under Optimal Separation
Furthermore, under a variant of the "no large sub-cluster'' condition from in prior work [BKK22], we show that our algorithm outputs an accurate clustering, not just a refinement, even for general-weight mixtures.
Near-Optimal Algorithms for Gaussians with Huber Contamination: Mean Estimation and Linear Regression
We study the fundamental problems of Gaussian mean estimation and linear regression with Gaussian covariates in the presence of Huber contamination.
Testing Closeness of Multivariate Distributions via Ramsey Theory
Our main result is the first closeness tester for this problem with {\em sub-learning} sample complexity in any fixed dimension and a nearly-matching sample complexity lower bound.
Online Robust Mean Estimation
We note that if the algorithm is allowed to wait until time $T$ to report its estimate, this reduces to the well-studied problem of robust mean estimation.
Distribution-Independent Regression for Generalized Linear Models with Oblivious Corruptions
Our goal is to accurately recover a \new{parameter vector $w$ such that the} function $g(w \cdot x)$ \new{has} arbitrarily small error when compared to the true values $g(w^* \cdot x)$, rather than the noisy measurements $y$.
Self-Directed Linear Classification
In contrast, under a worst- or random-ordering, the number of mistakes must be at least $\Omega(d \log n)$, even when the points are drawn uniformly from the unit sphere and the learner only needs to predict the labels for $1\%$ of them.
Efficiently Learning One-Hidden-Layer ReLU Networks via Schur Polynomials
Our main result is an efficient algorithm for this learning task with sample and computational complexity $(dk/\epsilon)^{O(k)}$, where $\epsilon>0$ is the target accuracy.
Information-Computation Tradeoffs for Learning Margin Halfspaces with Random Classification Noise
Our main result is a lower bound for Statistical Query (SQ) algorithms and low-degree polynomial tests suggesting that the quadratic dependence on $1/\epsilon$ in the sample complexity is inherent for computationally efficient algorithms.
SQ Lower Bounds for Learning Bounded Covariance GMMs
In the special case where the separation is on the order of $k^{1/2}$, we additionally obtain fine-grained SQ lower bounds with the correct exponent.
Robustly Learning a Single Neuron via Sharpness
We study the problem of learning a single neuron with respect to the $L_2^2$-loss in the presence of adversarial label noise.
Nearly-Linear Time and Streaming Algorithms for Outlier-Robust PCA
Our main contribution is to develop a nearly-linear time algorithm for robust PCA with near-optimal error guarantees.
Near-Optimal Cryptographic Hardness of Agnostically Learning Halfspaces and ReLU Regression under Gaussian Marginals
We study the task of agnostically learning halfspaces under the Gaussian distribution.
A Nearly Tight Bound for Fitting an Ellipsoid to Gaussian Random Points
We prove that for $c>0$ a sufficiently small universal constant that a random set of $c d^2/\log^4(d)$ independent Gaussian random points in $\mathbb{R}^d$ lie on a common ellipsoid with high probability.
A Strongly Polynomial Algorithm for Approximate Forster Transforms and its Application to Halfspace Learning
By leveraging our strongly polynomial Forster algorithm, we obtain the first strongly polynomial time algorithm for {\em distribution-free} PAC learning of halfspaces.
Outlier-Robust Sparse Mean Estimation for Heavy-Tailed Distributions
We study the fundamental task of outlier-robust mean estimation for heavy-tailed distributions in the presence of sparsity.
Gaussian Mean Testing Made Simple
Here we give an extremely simple algorithm for Gaussian mean testing with a one-page analysis.
SQ Lower Bounds for Learning Single Neurons with Massart Noise
We study the problem of PAC learning a single neuron in the presence of Massart noise.
Cryptographic Hardness of Learning Halfspaces with Massart Noise
We study the complexity of PAC learning halfspaces in the presence of Massart noise.
Near-Optimal Bounds for Testing Histogram Distributions
We investigate the problem of testing whether a discrete probability distribution over an ordered domain is a histogram on a specified number of bins.
Learning a Single Neuron with Adversarial Label Noise via Gradient Descent
For the ReLU activation, we give an efficient algorithm with sample complexity $\tilde{O}(d\, \polylog(1/\epsilon))$.
List-Decodable Sparse Mean Estimation via Difference-of-Pairs Filtering
We study the problem of list-decodable sparse mean estimation.
Optimal SQ Lower Bounds for Robustly Learning Discrete Product Distributions and Ising Models
We establish optimal Statistical Query (SQ) lower bounds for robustly learning certain families of discrete high-dimensional distributions.
Robust Sparse Mean Estimation via Sum of Squares
In this work, we develop the first efficient algorithms for robust sparse mean estimation without a priori knowledge of the covariance.
Streaming Algorithms for High-Dimensional Robust Statistics
In this work, we develop the first efficient streaming algorithms for high-dimensional robust statistics with near-optimal memory requirements (up to logarithmic factors).
Non-Gaussian Component Analysis via Lattice Basis Reduction
Non-Gaussian Component Analysis (NGCA) is the following distribution learning problem: Given i. i. d.
Outlier-Robust Sparse Estimation via Non-Convex Optimization
We explore the connection between outlier-robust high-dimensional statistics and non-convex optimization in the presence of sparsity constraints, with a focus on the fundamental tasks of robust sparse mean estimation and robust sparse PCA.
ReLU Regression with Massart Noise
This supervised learning task is efficiently solvable in the realizable setting, but is known to be computationally hard with adversarial label noise.
Learning General Halfspaces with General Massart Noise under the Gaussian Distribution
We study the general problem and establish the following: For $\eta <1/2$, we give a learning algorithm for general halfspaces with sample and computational complexity $d^{O_{\eta}(\log(1/\gamma))}\mathrm{poly}(1/\epsilon)$, where $\gamma =\max\{\epsilon, \min\{\mathbf{Pr}[f(\mathbf{x}) = 1], \mathbf{Pr}[f(\mathbf{x}) = -1]\} \}$ is the bias of the target halfspace $f$.
Forster Decomposition and Learning Halfspaces with Noise
A Forster transform is an operation that turns a distribution into one with good anti-concentration properties.
Statistical Query Lower Bounds for List-Decodable Linear Regression
We study the problem of list-decodable linear regression, where an adversary can corrupt a majority of the examples.
Clustering Mixture Models in Almost-Linear Time via List-Decodable Mean Estimation
We leverage this result, together with additional techniques, to obtain the first almost-linear time algorithms for clustering mixtures of $k$ separated well-behaved distributions, nearly-matching the statistical guarantees of spectral methods.
Boosting in the Presence of Massart Noise
Our upper and lower bounds characterize the complexity of boosting in the distribution-independent PAC model with Massart noise.
Agnostic Proper Learning of Halfspaces under Gaussian Marginals
We study the problem of agnostically learning halfspaces under the Gaussian distribution.
The Optimality of Polynomial Regression for Agnostic Learning under Gaussian Marginals
We study the problem of agnostic learning under the Gaussian distribution.
Outlier-Robust Learning of Ising Models Under Dobrushin's Condition
We study the problem of learning Ising models satisfying Dobrushin's condition in the outlier-robust setting where a constant fraction of the samples are adversarially corrupted.
The Sample Complexity of Robust Covariance Testing
This lower bound is best possible, as $O(d^2)$ samples suffice to even robustly {\em learn} the covariance.
Near-Optimal Statistical Query Hardness of Learning Halfspaces with Massart Noise
The best known $\mathrm{poly}(d, 1/\epsilon)$-time algorithms for this problem achieve error of $\eta+\epsilon$, which can be far from the optimal bound of $\mathrm{OPT}+\epsilon$, where $\mathrm{OPT} = \mathbf{E}_{x \sim D_x} [\eta(x)]$.
Small Covers for Near-Zero Sets of Polynomials and Learning Latent Variable Models
Our result is constructive yielding an algorithm to compute such an $\epsilon$-cover that runs in time $\mathrm{poly}(M)$.
Robustly Learning Mixtures of $k$ Arbitrary Gaussians
We give a polynomial-time algorithm for the problem of robustly estimating a mixture of $k$ arbitrary Gaussians in $\mathbb{R}^d$, for any fixed $k$, in the presence of a constant fraction of arbitrary corruptions.
List-Decodable Mean Estimation in Nearly-PCA Time
Our algorithm runs in time $\widetilde{O}(ndk)$ for all $k = O(\sqrt{d}) \cup \Omega(d)$, where $n$ is the size of the dataset.
A Polynomial Time Algorithm for Learning Halfspaces with Tsybakov Noise
{\em We give the first polynomial-time algorithm for this fundamental learning problem.}
Optimal Testing of Discrete Distributions with High Probability
To illustrate the generality of our methods, we give optimal algorithms for testing collections of distributions and testing closeness with unequal sized samples.
The Complexity of Adversarially Robust Proper Learning of Halfspaces with Agnostic Noise
We study the computational complexity of adversarially robust proper learning of halfspaces in the distribution-independent agnostic PAC model, with a focus on $L_p$ perturbations.
Outlier Robust Mean Estimation with Subgaussian Rates via Stability
We study the problem of outlier robust high-dimensional mean estimation under a finite covariance assumption, and more broadly under finite low-degree moment assumptions.
Near-Optimal SQ Lower Bounds for Agnostically Learning Halfspaces and ReLUs under Gaussian Marginals
We study the fundamental problems of agnostically learning halfspaces and ReLUs under Gaussian marginals.
Algorithms and SQ Lower Bounds for PAC Learning One-Hidden-Layer ReLU Networks
For the case of positive coefficients, we give the first polynomial-time algorithm for this learning problem for $k$ up to $\tilde{O}(\sqrt{\log d})$.
List-Decodable Mean Estimation via Iterative Multi-Filtering
We study the problem of {\em list-decodable mean estimation} for bounded covariance distributions.
Non-Convex SGD Learns Halfspaces with Adversarial Label Noise
We study the problem of agnostically learning homogeneous halfspaces in the distribution-specific PAC model.
Learning Halfspaces with Tsybakov Noise
In the Tsybakov noise model, each label is independently flipped with some probability which is controlled by an adversary.
Approximation Schemes for ReLU Regression
We consider the fundamental problem of ReLU regression, where the goal is to output the best fitting ReLU with respect to square loss given access to draws from some unknown distribution.
Efficiently Learning Adversarially Robust Halfspaces with Noise
We study the problem of learning adversarially robust halfspaces in the distribution-independent setting.
Robustly Learning any Clusterable Mixture of Gaussians
The key ingredients of this proof are a novel use of SoS-certifiable anti-concentration and a new characterization of pairs of Gaussians with small (dimension-independent) overlap in terms of their parameter distance.
High-Dimensional Robust Mean Estimation via Gradient Descent
We study the problem of high-dimensional robust mean estimation in the presence of a constant fraction of adversarial outliers.
Efficient Algorithms for Multidimensional Segmented Regression
We study the fundamental problem of fixed design {\em multidimensional segmented regression}: Given noisy samples from a function $f$, promised to be piecewise linear on an unknown set of $k$ rectangles, we want to recover $f$ up to a desired accuracy in mean-squared error.
Learning Halfspaces with Massart Noise Under Structured Distributions
We study the problem of learning halfspaces with Massart noise in the distribution-specific PAC model.
Private Testing of Distributions via Sample Permutations
In this paper, we use the framework of property testing to design algorithms to test the properties of the distribution that the data is drawn from with respect to differential privacy.
Outlier-Robust High-Dimensional Sparse Estimation via Iterative Filtering
Specifically, we focus on the fundamental problems of robust sparse mean estimation and robust sparse PCA.
Recent Advances in Algorithmic High-Dimensional Robust Statistics
Learning in the presence of outliers is a fundamental problem in statistics.
Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a Margin
We study the problem of {\em properly} learning large margin halfspaces in the agnostic PAC model.
Distribution-Independent PAC Learning of Halfspaces with Massart Noise
The goal is to find a hypothesis $h$ that minimizes the misclassification error $\mathbf{Pr}_{(\mathbf{x}, y) \sim \mathcal{D}} \left[ h(\mathbf{x}) \neq y \right]$.
Communication and Memory Efficient Testing of Discrete Distributions
We study distribution testing with communication and memory constraints in the following computational models: (1) The {\em one-pass streaming model} where the goal is to minimize the sample complexity of the protocol subject to a memory constraint, and (2) A {\em distributed model} where the data samples reside at multiple machines and the goal is to minimize the communication cost of the protocol.
Faster Algorithms for High-Dimensional Robust Covariance Estimation
We study the problem of estimating the covariance matrix of a high-dimensional distribution when a small constant fraction of the samples can be arbitrarily corrupted.
Equipping Experts/Bandits with Long-term Memory
We propose the first reduction-based approach to obtaining long-term memory guarantees for online learning in the sense of Bousquet and Warmuth, 2002, by reducing the problem to achieving typical switching regret.
On the Complexity of the Inverse Semivalue Problem for Weighted Voting Games
In this work, we study the computational complexity of the inverse problem when the power index belongs to the class of semivalues.
Testing for Families of Distributions via the Fourier Transform
We study the general problem of testing whether an unknown discrete distribution belongs to a specified family of distributions.
High-Dimensional Robust Mean Estimation in Nearly-Linear Time
We study the fundamental problem of high-dimensional mean estimation in a robust model where a constant fraction of the samples are adversarially corrupted.
Degree-$d$ Chow Parameters Robustly Determine Degree-$d$ PTFs (and Algorithmic Applications)
Our robust identifiability result gives the following algorithmic applications: First, we show that Boolean degree-$d$ PTFs can be efficiently approximately reconstructed from approximations to their degree-$d$ Chow parameters.
Differentially Private Identity and Equivalence Testing of Discrete Distributions
Our theoretical results significantly improve over the best known algorithms for identity testing, and are the first results for private equivalence testing.
Efficient Algorithms and Lower Bounds for Robust Linear Regression
An error of $\Omega (\epsilon \sigma)$ is information-theoretically necessary, even with infinite sample size.
Testing Identity of Multidimensional Histograms
We give the first identity tester for this problem with {\em sub-learning} sample complexity in any fixed dimension and a nearly-matching sample complexity lower bound.
Sever: A Robust Meta-Algorithm for Stochastic Optimization
In high dimensions, most machine learning methods are brittle to even a small fraction of structured outliers.
Near-Optimal Sample Complexity Bounds for Maximum Likelihood Estimation of Multivariate Log-concave Densities
Prior to this work, no finite sample upper bound was known for this estimator in more than $3$ dimensions.
Fast and Sample Near-Optimal Algorithms for Learning Multidimensional Histograms
We give an algorithm for this learning problem that uses $n = \tilde{O}_d(k/\epsilon^2)$ samples and runs in time $\tilde{O}_d(n)$.
Communication-Efficient Distributed Learning of Discrete Distributions
For the case of structured distributions, such as k-histograms and monotone distributions, we design distributed learning algorithms that achieve significantly better communication guarantees than the naive ones, and obtain tight upper and lower bounds in several regimes.
List-Decodable Robust Mean Estimation and Learning Mixtures of Spherical Gaussians
We give a learning algorithm for mixtures of spherical Gaussians that succeeds under significantly weaker separation assumptions compared to prior work.
Sharp Bounds for Generalized Uniformity Testing
We study the problem of generalized uniformity testing \cite{BC17} of a discrete probability distribution: Given samples from a probability distribution $p$ over an {\em unknown} discrete domain $\mathbf{\Omega}$, we want to distinguish, with probability at least $2/3$, between the case that $p$ is uniform on some {\em subset} of $\mathbf{\Omega}$ versus $\epsilon$-far, in total variation distance, from any such uniform distribution.
Optimal Identity Testing with High Probability
Our new upper and lower bounds show that the optimal sample complexity of identity testing is \[ \Theta\left( \frac{1}{\epsilon^2}\left(\sqrt{n \log(1/\delta)} + \log(1/\delta) \right)\right) \] for any $n, \varepsilon$, and $\delta$.
Differentially Private Identity and Closeness Testing of Discrete Distributions
We investigate the problems of identity and closeness testing over a discrete population from random samples.
Learning Geometric Concepts with Nasty Noise
We give the first polynomial-time PAC learning algorithms for these concept classes with dimension-independent error guarantees in the presence of nasty noise under the Gaussian distribution.
Robustly Learning a Gaussian: Getting Optimal Error, Efficiently
We give robust estimators that achieve estimation error $O(\varepsilon)$ in the total variation distance, which is optimal up to a universal constant that is independent of the dimension.
Near-Optimal Closeness Testing of Discrete Histogram Distributions
Given a set of samples from two $k$-histogram distributions $p, q$ over $[n]$, we want to distinguish (with high probability) between the cases that $p = q$ and $\|p-q\|_1 \geq \epsilon$.
Being Robust (in High Dimensions) Can Be Practical
Robust estimation is much more challenging in high dimensions than it is in one dimension: Most techniques either lead to intractable optimization problems or estimators that can tolerate only a tiny fraction of errors.
Testing Bayesian Networks
This work initiates a systematic investigation of testing high-dimensional structured distributions by focusing on testing Bayesian networks -- the prototypical family of directed graphical models.
Collision-based Testers are Optimal for Uniformity and Closeness
We study the fundamental problems of (i) uniformity testing of a discrete distribution, and (ii) closeness testing between two discrete distributions with bounded $\ell_2$-norm.
Statistical Query Lower Bounds for Robust Estimation of High-dimensional Gaussians and Gaussian Mixtures
For each of these problems, we show a {\em super-polynomial gap} between the (information-theoretic) sample complexity and the computational complexity of {\em any} Statistical Query algorithm for the problem.
Fast Algorithms for Segmented Regression
We study the fixed design segmented regression problem: Given noisy samples from a piecewise linear function $f$, we want to recover $f$ up to a desired accuracy in mean-squared error.
Robust Learning of Fixed-Structure Bayesian Networks
We investigate the problem of learning Bayesian networks in a robust model where an $\epsilon$-fraction of the samples are adversarially corrupted.
Efficient Robust Proper Learning of Log-concave Distributions
We study the {\em robust proper learning} of univariate log-concave distributions (over continuous and discrete domains).
Learning Multivariate Log-concave Distributions
Prior to our work, no upper bound on the sample complexity of this learning problem was known for the case of $d>3$.
Robust Estimators in High Dimensions without the Computational Intractability
We study high-dimensional distribution learning in an agnostic setting where an adversary is allowed to arbitrarily corrupt an $\varepsilon$-fraction of the samples.
Differentially Private Learning of Structured Discrete Distributions
We investigate the problem of learning an unknown probability distribution over a discrete population from random samples.
Properly Learning Poisson Binomial Distributions in Almost Polynomial Time
Given $\widetilde{O}(1/\epsilon^2)$ samples from an unknown PBD $\mathbf{p}$, our algorithm runs in time $(1/\epsilon)^{O(\log \log (1/\epsilon))}$, and outputs a hypothesis PBD that is $\epsilon$-close to $\mathbf{p}$ in total variation distance.
The Fourier Transform of Poisson Multinomial Distributions and its Algorithmic Applications
An $(n, k)$-Poisson Multinomial Distribution (PMD) is a random variable of the form $X = \sum_{i=1}^n X_i$, where the $X_i$'s are independent random vectors supported on the set of standard basis vectors in $\mathbb{R}^k.$ In this paper, we obtain a refined structural understanding of PMDs by analyzing their Fourier transform.
Sample-Optimal Density Estimation in Nearly-Linear Time
Let $f$ be the density function of an arbitrary univariate distribution, and suppose that $f$ is $\mathrm{OPT}$-close in $L_1$-distance to an unknown piecewise polynomial function with $t$ interval pieces and degree $d$.
Optimal Learning via the Fourier Transform for Sums of Independent Integer Random Variables
As one of our main structural contributions, we give an efficient algorithm to construct a sparse {\em proper} $\epsilon$-cover for ${\cal S}_{n, k},$ in total variation distance.
Near-Optimal Density Estimation in Near-Linear Time Using Variable-Width Histograms
The "approximation factor" $C$ in our result is inherent in the problem, as we prove that no algorithm with sample size bounded in terms of $k$ and $\epsilon$ can achieve $C<2$ regardless of what kind of hypothesis distribution it uses.
Optimal Algorithms for Testing Closeness of Discrete Distributions
We study the question of closeness testing for two discrete distributions.
Efficient Density Estimation via Piecewise Polynomial Approximation
We give an algorithm that draws $\tilde{O}(t\new{(d+1)}/\eps^2)$ samples from $p$, runs in time $\poly(t, d, 1/\eps)$, and with high probability outputs a piecewise polynomial hypothesis distribution $h$ that is $(O(\tau)+\eps)$-close (in total variation distance) to $p$.
Inverse problems in approximate uniform generation
In such an inverse problem, the algorithm is given uniform random satisfying assignments of an unknown function $f$ belonging to a class $\C$ of Boolean functions, and the goal is to output a probability distribution $D$ which is $\epsilon$-close, in total variation distance, to the uniform distribution over $f^{-1}(1)$.
Learning Poisson Binomial Distributions
Our second main result is a {\em proper} learning algorithm that learns to $\eps$-accuracy using $\tilde{O}(1/\eps^2)$ samples, and runs in time $(1/\eps)^{\poly (\log (1/\eps))} \cdot \log n$.
Learning $k$-Modal Distributions via Testing
The learning algorithm is given access to independent samples drawn from an unknown $k$-modal distribution $p$, and it must output a hypothesis distribution $\widehat{p}$ such that with high probability the total variation distance between $p$ and $\widehat{p}$ is at most $\epsilon.$ Our main goal is to obtain \emph{computationally efficient} algorithms for this problem that use (close to) an information-theoretically optimal number of samples.
