Search Results for author:
Found 75 papers, 11 papers with code
Do Transformers Parse while Predicting the Masked Word?
We also show that the Inside-Outside algorithm is optimal for masked language modeling loss on the PCFG-generated data.
Hiding Data Helps: On the Benefits of Masking for Sparse Coding
Furthermore, drawing from the growing body of work on self-supervised learning, we propose a novel masking objective and we prove that minimizing this new objective can recover the ground-truth dictionary.
Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regression
In this work, we give a different parametrization of the model which leads to a new implicit regularization effect that combines the benefit of $\ell_1$ and $\ell_2$ interpolators.
Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup
Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels.
Understanding Edge-of-Stability Training Dynamics with a Minimalist Example
Globally we observe that the training dynamics for our example has an interesting bifurcating behavior, which was also observed in the training of neural nets.
Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networks
Monotonic linear interpolation (MLI) - on the line connecting a random initialization with the minimizer it converges to, the loss and accuracy are monotonic - is a phenomenon that is commonly observed in the training of neural networks.
Customizing ML Predictions for Online Algorithms
A popular line of recent research incorporates ML advice in the design of online algorithms to improve their performance in typical instances.
A Regression Approach to Learning-Augmented Online Algorithms
The emerging field of learning-augmented online algorithms uses ML techniques to predict future input parameters and thereby improve the performance of online algorithms.
Online Algorithms with Multiple Predictions
In this paper, we give a generic algorithmic framework for online covering problems with multiple predictions that obtains an online solution that is competitive against the performance of the best predictor.
Understanding The Robustness of Self-supervised Learning Through Topic Modeling
Self-supervised learning has significantly improved the performance of many NLP tasks.
Towards Understanding the Data Dependency of Mixup-style Training
Despite seeing very few true data points during training, models trained using Mixup seem to still minimize the original empirical risk and exhibit better generalization and robustness on various tasks when compared to standard training.
One Objective for All Models --- Self-supervised Learning for Topic Models
Self-supervised learning has significantly improved the performance of many NLP tasks.
Outlier-Robust Sparse Estimation via Non-Convex Optimization
We explore the connection between outlier-robust high-dimensional statistics and non-convex optimization in the presence of sparsity constraints, with a focus on the fundamental tasks of robust sparse mean estimation and robust sparse PCA.
Understanding Deflation Process in Over-parametrized Tensor Decomposition
In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems.
A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network
We show that as long as the loss is already lower than a threshold (polynomial in relevant parameters), all student neurons in an over-parameterized two-layer neural network will converge to one of teacher neurons, and the loss will go to 0.
Beyond Lazy Training for Over-parameterized Tensor Decomposition
We show that in a lazy training regime (similar to the NTK regime for neural networks) one needs at least $m = \Omega(d^{l-1})$, while a variant of gradient descent can find an approximate tensor when $m = O^*(r^{2. 5l}\log d)$.
Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
We can analyze the properties of these smaller matrices and prove the structure of top eigenspace random 2-layer networks.
Efficient sampling from the Bingham distribution
We give a algorithm for exact sampling from the Bingham distribution $p(x)\propto \exp(x^\top A x)$ on the sphere $\mathcal S^{d-1}$ with expected runtime of $\operatorname{poly}(d, \lambda_{\max}(A)-\lambda_{\min}(A))$.
Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
Solving this problem using a learning-to-learn approach -- using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates -- was recently shown to be effective.
Optimization Landscape of Tucker Decomposition
Finding a Tucker decomposition is a nonconvex optimization problem.
Extracting Latent State Representations with Linear Dynamics from Rich Observations
In this work we study a model where there is a hidden linear subspace in which the dynamics is linear.
Energy-Aware DNN Graph Optimization
Unlike existing work in deep neural network (DNN) graphs optimization for inference performance, we explore DNN graph optimization for energy awareness and savings for power- and resource-constrained machine learning devices.
High-Dimensional Robust Mean Estimation via Gradient Descent
We study the problem of high-dimensional robust mean estimation in the presence of a constant fraction of adversarial outliers.
Spectral Learning on Matrices and Tensors
PCA and other spectral techniques applied to matrices have several limitations.
Estimating Normalizing Constants for Log-Concave Distributions: Algorithms and Lower Bounds
Estimating the normalizing constant of an unnormalized probability distribution has important applications in computer science, statistical physics, machine learning, and statistics.
Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
It has been observed \citep{zhang2016understanding} that deep neural networks can memorize: they achieve 100\% accuracy on training data.
Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
Mode connectivity is a surprising phenomenon in the loss landscape of deep nets.
Faster Algorithms for High-Dimensional Robust Covariance Estimation
We study the problem of estimating the covariance matrix of a high-dimensional distribution when a small constant fraction of the samples can be arbitrarily corrupted.
Stabilized SVRG: Simple Variance Reduction for Nonconvex Optimization
Variance reduction techniques like SVRG provide simple and fast algorithms for optimizing a convex finite-sum objective.
Rethinking learning rate schedules for stochastic optimization
One plausible explanation is that non-convex neural network training procedures are better suited to the use of fundamentally different learning rate schedules, such as the ``cut the learning rate every constant number of epochs'' method (which more closely resembles an exponentially decaying learning rate schedule); note that this widely used schedule is in stark contrast to the polynomial decay schemes prescribed in the stochastic approximation literature, which are indeed shown to be (worst case) optimal for classes of convex optimization problems.
The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares
First, this work shows that even if the time horizon T (i. e. the number of iterations SGD is run for) is known in advance, SGD's final iterate behavior with any polynomially decaying learning rate scheme is highly sub-optimal compared to the minimax rate (by a condition number factor in the strongly convex case and a factor of $\sqrt{T}$ in the non-strongly convex case).
On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial.
A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm
In this note, we derive concentration inequalities for random vectors with subGaussian norm (a generalization of both subGaussian random vectors and norm bounded random vectors), which are tight up to logarithmic factors.
Understanding Composition of Word Embeddings via Tensor Decomposition
Word embedding is a powerful tool in natural language processing.
Simulated Tempering Langevin Monte Carlo II: An Improved Proof using Soft Markov Chain Decomposition
Previous approaches rely on decomposing the state space as a partition of sets, while our approach can be thought of as decomposing the stationary measure as a mixture of distributions (a "soft partition").
High-Dimensional Robust Mean Estimation in Nearly-Linear Time
We study the fundamental problem of high-dimensional mean estimation in a robust model where a constant fraction of the samples are adversarially corrupted.
Learning Two-layer Neural Networks with Symmetric Inputs
We give a new algorithm for learning a two-layer neural network under a general class of input distributions.
Non-Convex Matrix Completion Against a Semi-Random Adversary
Matrix completion is a well-studied problem with many machine learning applications.
On the Local Minima of the Empirical Risk
Our objective is to find the $\epsilon$-approximate local minima of the underlying function $F$ while avoiding the shallow local minima---arising because of the tolerance $\nu$---which exist only in $f$.
Stronger generalization bounds for deep nets via a compression approach
Analysis of correctness of our compression relies upon some newly identified \textquotedblleft noise stability\textquotedblright properties of trained deep nets, which are also experimentally verified.
Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest 3) they inherently allow for richly parameterized policies.
Global Convergence of Policy Gradient Methods for Linearized Control Problems
Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model; 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest; 3) they inherently allow for richly parameterized policies.
Learning One-hidden-layer Neural Networks with Landscape Design
All global minima of $G$ correspond to the ground truth parameters.
Beyond Log-concavity: Provable Guarantees for Sampling Multi-modal Distributions using Simulated Tempering Langevin Monte Carlo
We analyze this Markov chain for the canonical multi-modal distribution: a mixture of gaussians (of equal variance).
On the Optimization Landscape of Tensor Decompositions
The landscape of many objective functions in learning has been conjectured to have the geometric property that "all local optima are (approximately) global optima", and thus they can be solved efficiently by local search algorithms.
No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
In this paper we develop a new framework that captures the common landscape underlying the common non-convex low-rank matrix problems including matrix sensing, matrix completion and robust PCA.
Generalization and Equilibrium in Generative Adversarial Nets (GANs)
We show that training of generative adversarial network (GAN) may not have good generalization properties; e. g., training may appear successful but the trained distribution may be far from target distribution in standard metrics.
How to Escape Saddle Points Efficiently
This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i. e., it is almost "dimension-free").
On the ability of neural nets to express distributions
We take a first cut at explaining the expressivity of multilayer nets by giving a sufficient criterion for a function to be approximable by a neural network with $n$ hidden layers.
Provable learning of Noisy-or Networks
Many machine learning applications use latent variable models to explain structure in data, whereby visible variables (= coordinates of the given datapoint) are explained as a probabilistic function of some hidden variables.
Homotopy Analysis for Tensor PCA
For the challenging problem of tensor PCA, we prove global convergence of the homotopy method in the "high noise" regime.
Provable Algorithms for Inference in Topic Models
But designing provable algorithms for inference has proven to be more challenging.
Matrix Completion has No Spurious Local Minimum
Matrix completion is a basic machine learning problem that has wide applications, especially in collaborative filtering and recommender systems.
Efficient Algorithms for Large-scale Generalized Eigenvector Computation and Canonical Correlation Analysis
Our algorithm is linear in the input size and the number of components $k$ up to a $\log(k)$ factor.
Efficient approaches for escaping higher order saddle points in non-convex optimization
Local search heuristics for non-convex optimizations are popular in applied machine learning.
Rich Component Analysis
In this paper, we develop the general framework of Rich Component Analysis (RCA) to model settings where the observations from different views are driven by different sets of latent components, and each component can be a complex, high-dimensional distribution.
Intersecting Faces: Non-negative Matrix Factorization With New Guarantees
A plethora of algorithms have been developed to tackle NMF, but due to the non-convex nature of the problem, there is little guarantee on how well these methods work.
Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization
We develop a family of accelerated stochastic algorithms that minimize sums of convex functions.
Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms
We also give a polynomial time algorithm for certifying the injective norm of random low rank tensors.
Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points.
Simple, Efficient, and Neural Algorithms for Sparse Coding
Its standard formulation is as a non-convex optimization problem which is solved in practice by heuristics based on alternating minimization.
Learning Mixtures of Gaussians in High Dimensions
Unfortunately, learning mixture of Gaussians is an information theoretically hard problem: in order to learn the parameters up to a reasonable accuracy, the number of samples required is exponential in the number of Gaussian components in the worst case.
Competing with the Empirical Risk Minimizer in a Single Pass
In the absence of computational constraints, the minimizer of a sample average of observed data -- commonly referred to as either the empirical risk minimizer (ERM) or the $M$-estimator -- is widely regarded as the estimation strategy of choice due to its desirable statistical convergence properties.
Minimal Realization Problems for Hidden Markov Models
Consider a stationary discrete random process with alphabet size d, which is assumed to be the output process of an unknown stationary Hidden Markov Model (HMM).
Analyzing Tensor Power Method Dynamics in Overcomplete Regime
We present a novel analysis of the dynamics of tensor power iterations in the overcomplete regime where the tensor CP rank is larger than the input dimension.
Sample Complexity Analysis for Learning Overcomplete Latent Variable Models through Tensor Methods
In the unsupervised setting, we use a simple initialization algorithm based on SVD of the tensor slices, and provide guarantees under the stricter condition that $k\le \beta d$ (where constant $\beta$ can be larger than $1$), where the tensor method recovers the components under a polynomial running time (and exponential in $\beta$).
Guaranteed Non-Orthogonal Tensor Decomposition via Alternating Rank-$1$ Updates
In this paper, we provide local and global convergence guarantees for recovering CP (Candecomp/Parafac) tensor decomposition.
More Algorithms for Provable Dictionary Learning
In dictionary learning, also known as sparse coding, the algorithm is given samples of the form $y = Ax$ where $x\in \mathbb{R}^m$ is an unknown random sparse vector and $A$ is an unknown dictionary matrix in $\mathbb{R}^{n\times m}$ (usually $m > n$, which is the overcomplete case).
Provable Bounds for Learning Some Deep Representations
The analysis of the algorithm reveals interesting structure of neural networks with random edge weights.
New Algorithms for Learning Incoherent and Overcomplete Dictionaries
In sparse recovery we are given a matrix $A$ (the dictionary) and a vector of the form $A X$ where $X$ is sparse, and the goal is to recover $X$.
A Tensor Approach to Learning Mixed Membership Community Models
We provide guaranteed recovery of community memberships and model parameters and present a careful finite sample analysis of our learning method.
A Practical Algorithm for Topic Modeling with Provable Guarantees
Topic models provide a useful method for dimensionality reduction and exploratory data analysis in large text corpora.
Provable ICA with Unknown Gaussian Noise, with Implications for Gaussian Mixtures and Autoencoders
We present a new algorithm for Independent Component Analysis (ICA) which has provable performance guarantees.
Tensor decompositions for learning latent variable models
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order).
Learning Topic Models - Going beyond SVD
Topic Modeling is an approach used for automatic comprehension and classification of data in a variety of settings, and perhaps the canonical application is in uncovering thematic structure in a corpus of documents.
