Search Results for author:
Found 203 papers, 48 papers with code
Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimization
We provide a unified analysis of two-timescale gradient descent ascent (TTGDA) for solving structured nonconvex minimax optimization problems in the form of $\min_\textbf{x} \max_{\textbf{y} \in Y} f(\textbf{x}, \textbf{y})$, where the objective function $f(\textbf{x}, \textbf{y})$ is nonconvex in $\textbf{x}$ and concave in $\textbf{y}$, and the constraint set $Y \subseteq \mathbb{R}^n$ is convex and bounded.
Perseus: A Simple and Optimal High-Order Method for Variational Inequalities
Our method with restarting attains a linear rate for smooth and uniformly monotone VIs and a local superlinear rate for smooth and strongly monotone VIs.
Uncertainty Sets for Image Classifiers using Conformal Prediction
Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings.
Learning from eXtreme Bandit Feedback
In POXM, the selected actions for the sIS estimator are the top-p actions of the logging policy, where p is adjusted from the data and is significantly smaller than the size of the action space.
Extreme Multi-Label Classification
MUlTI-LABEL-ClASSIFICATION
+1
Exploration in two-stage recommender systems
In this work, we focus on the complementary issue of exploration.
On Localized Discrepancy for Domain Adaptation
Finally, we further extend the localized discrepancies for achieving super transfer and derive generalization bounds that could be even more sample-efficient on source domain.
Transferable Calibration with Lower Bias and Variance in Domain Adaptation
In this paper, we delve into the open problem of Calibration in DA, which is extremely challenging due to the coexistence of domain shift and the lack of target labels.
Optimal Robust Linear Regression in Nearly Linear Time
We study the problem of high-dimensional robust linear regression where a learner is given access to $n$ samples from the generative model $Y = \langle X, w^* \rangle + \epsilon$ (with $X \in \mathbb{R}^d$ and $\epsilon$ independent), in which an $\eta$ fraction of the samples have been adversarially corrupted.
Manifold Learning via Manifold Deflation
Nonlinear dimensionality reduction methods provide a valuable means to visualize and interpret high-dimensional data.
Accelerated Message Passing for Entropy-Regularized MAP Inference
Maximum a posteriori (MAP) inference in discrete-valued Markov random fields is a fundamental problem in machine learning that involves identifying the most likely configuration of random variables given a distribution.
On Projection Robust Optimal Transport: Sample Complexity and Model Misspecification
Yet, the behavior of minimum Wasserstein estimators is poorly understood, notably in high-dimensional regimes or under model misspecification.
On the Theory of Transfer Learning: The Importance of Task Diversity
Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j \circ h$ in a general function class $\mathcal{F} \circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$.
Active Learning for Nonlinear System Identification with Guarantees
While the identification of nonlinear dynamical systems is a fundamental building block of model-based reinforcement learning and feedback control, its sample complexity is only understood for systems that either have discrete states and actions or for systems that can be identified from data generated by i. i. d.
Projection Robust Wasserstein Distance and Riemannian Optimization
Projection robust Wasserstein (PRW) distance, or Wasserstein projection pursuit (WPP), is a robust variant of the Wasserstein distance.
Instability, Computational Efficiency and Statistical Accuracy
Many statistical estimators are defined as the fixed point of a data-dependent operator, with estimators based on minimizing a cost function being an important special case.
Lower bounds in multiple testing: A framework based on derandomized proxies
A line of more recent work in multiple testing has begun to investigate the tradeoffs between the FDR and FNR and to provide lower bounds on the performance of procedures that depend on the model structure.
VCG Mechanism Design with Unknown Agent Values under Stochastic Bandit Feedback
To that end, we first define three notions of regret for the welfare, the individual utilities of each agent and that of the mechanism.
On dissipative symplectic integration with applications to gradient-based optimization
More specifically, we show that a generalization of symplectic integrators to nonconservative and in particular dissipative Hamiltonian systems is able to preserve rates of convergence up to a controlled error.
On Learning Rates and Schrödinger Operators
In this paper, we present a general theoretical analysis of the effect of the learning rate in stochastic gradient descent (SGD).
On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration
When the matrix $\bar{A}$ is Hurwitz, we prove a central limit theorem (CLT) for the averaged iterates with fixed step size and number of iterations going to infinity.
Identifying and Correcting Bias from Time- and Severity- Dependent Reporting Rates in the Estimation of the COVID-19 Case Fatality Rate
As we are in the middle of an active outbreak, estimating this measure will necessarily involve correcting for time- and severity- dependent reporting of cases, and time-lags in observed patient outcomes.
Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis
We address the problem of policy evaluation in discounted Markov decision processes, and provide instance-dependent guarantees on the $\ell_\infty$-error under a generative model.
Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information
Observational data are often accompanied by natural structural indices, such as time stamps or geographic locations, which are meaningful to prediction tasks but are often discarded.
Robustness Guarantees for Mode Estimation with an Application to Bandits
Mode estimation is a classical problem in statistics with a wide range of applications in machine learning.
Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives
We analyze the convergence rate of various momentum-based optimization algorithms from a dynamical systems point of view.
Provable Meta-Learning of Linear Representations
In this paper, we focus on the problem of multi-task linear regression -- in which multiple linear regression models share a common, low-dimensional linear representation.
Finite-Time Last-Iterate Convergence for Multi-Agent Learning in Games
In this paper, we consider multi-agent learning via online gradient descent in a class of games called $\lambda$-cocoercive games, a fairly broad class of games that admits many Nash equilibria and that properly includes unconstrained strongly monotone games.
On Thompson Sampling with Langevin Algorithms
The resulting approximate Thompson sampling algorithm has logarithmic regret and its computational complexity does not scale with the time horizon of the algorithm.
Robust Optimization for Fairness with Noisy Protected Groups
Second, we introduce two new approaches using robust optimization that, unlike the naive approach of only relying on $\hat{G}$, are guaranteed to satisfy fairness criteria on the true protected groups G while minimizing a training objective.
Decision-Making with Auto-Encoding Variational Bayes
To make decisions based on a model fit with auto-encoding variational Bayes (AEVB), practitioners often let the variational distribution serve as a surrogate for the posterior distribution.
Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization
Adaptivity is an important yet under-studied property in modern optimization theory.
Fixed-Support Wasserstein Barycenters: Computational Hardness and Fast Algorithm
We study the fixed-support Wasserstein barycenter problem (FS-WBP), which consists in computing the Wasserstein barycenter of $m$ discrete probability measures supported on a finite metric space of size $n$.
Continuous-time Lower Bounds for Gradient-based Algorithms
This article derives lower bounds on the convergence rate of continuous-time gradient-based optimization algorithms.
Optimization and Control Systems and Control Systems and Control
Near-Optimal Algorithms for Minimax Optimization
This paper presents the first algorithm with $\tilde{O}(\sqrt{\kappa_{\mathbf x}\kappa_{\mathbf y}})$ gradient complexity, matching the lower bound up to logarithmic factors.
Variance Reduction with Sparse Gradients
Variance reduction methods such as SVRG and SpiderBoost use a mixture of large and small batch gradients to reduce the variance of stochastic gradients.
Sampling for Bayesian Mixture Models: MCMC with Polynomial-Time Mixing
We study the problem of sampling from the power posterior distribution in Bayesian Gaussian mixture models, a robust version of the classical posterior.
Transferable Normalization: Towards Improving Transferability of Deep Neural Networks
Deep neural networks (DNNs) excel at learning representations when trained on large-scale datasets.
The Power of Batching in Multiple Hypothesis Testing
One important partition of algorithms for controlling the false discovery rate (FDR) in multiple testing is into offline and online algorithms.
On the Complexity of Approximating Multimarginal Optimal Transport
This provides a first \textit{near-linear time} complexity bound guarantee for approximating the MOT problem and matches the best known complexity bound for the Sinkhorn algorithm in the classical OT setting when $m = 2$.
Towards Understanding the Transferability of Deep Representations
3) The feasibility of transferability is related to the similarity of both input and label.
High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithm
We propose a Markov chain Monte Carlo (MCMC) algorithm based on third-order Langevin dynamics for sampling from distributions with log-concave and smooth densities.
How Does Learning Rate Decay Help Modern Neural Networks?
Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex.
A Higher-Order Swiss Army Infinitesimal Jackknife
The first-order approximation is known as the "infinitesimal jackknife" in the statistics literature and has been the subject of recent interest in machine learning for approximate CV.
Bayesian Robustness: A Nonasymptotic Viewpoint
We study the problem of robustly estimating the posterior distribution for the setting where observed data can be contaminated with potentially adversarial outliers.
Provably Efficient Reinforcement Learning with Linear Function Approximation
Modern Reinforcement Learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy.
Convergence Rates for Gaussian Mixtures of Experts
We provide a theoretical treatment of over-specified Gaussian mixtures of experts with covariate-free gating networks.
Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Games
In such games the state and action spaces are continuous and global Nash equilibria can be found be solving coupled Ricatti equations.
Stochastic Gradient and Langevin Processes
We prove quantitative convergence rates at which discrete Langevin-like processes converge to the invariant distribution of a related stochastic differential equation.
Convergence Rates of Smooth Message Passing with Rounding in Entropy-Regularized MAP Inference
Maximum a posteriori (MAP) inference is a fundamental computational paradigm for statistical inference.
Competing Bandits in Matching Markets
Stable matching, a classical model for two-sided markets, has long been studied with little consideration for how each side's preferences are learned.
Learning to Score Behaviors for Guided Policy Optimization
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
ML-LOO: Detecting Adversarial Examples with Feature Attribution
Furthermore, we extend our method to include multi-layer feature attributions in order to tackle the attacks with mixed confidence levels.
On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
We consider nonconvex-concave minimax problems, $\min_{\mathbf{x}} \max_{\mathbf{y} \in \mathcal{Y}} f(\mathbf{x}, \mathbf{y})$, where $f$ is nonconvex in $\mathbf{x}$ but concave in $\mathbf{y}$ and $\mathcal{Y}$ is a convex and bounded set.
Generalized Momentum-Based Methods: A Hamiltonian Perspective
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces.
On the Efficiency of Entropic Regularized Algorithms for Optimal Transport
We prove that APDAMD achieves the complexity bound of $\widetilde{O}(n^2\sqrt{\delta}\varepsilon^{-1})$ in which $\delta>0$ stands for the regularity of $\phi$.
Langevin Monte Carlo without smoothness
Langevin Monte Carlo (LMC) is an iterative algorithm used to generate samples from a distribution that is known only up to a normalizing constant.
Fast Algorithms for Computational Optimal Transport and Wasserstein Barycenter
First, we introduce the \emph{accelerated primal-dual randomized coordinate descent} (APDRCD) algorithm for computing the OT distance.
A Dynamical Systems Perspective on Nesterov Acceleration
We present a dynamical system framework for understanding Nesterov's accelerated gradient method.
A joint model of unpaired data from scRNA-seq and spatial transcriptomics for imputing missing gene expression measurements
Building upon domain adaptation work, we propose gimVI, a deep generative model for the integration of spatial transcriptomic data and scRNA-seq data that can be used to impute missing genes.
Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning
The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs–the Rendering Path Normalization (RPN).
On Structured Filtering-Clustering: Global Error Bound and Optimal First-Order Algorithms
We also conduct experiments on real datasets and the numerical results demonstrate the effectiveness of our algorithms.
Bridging Theory and Algorithm for Domain Adaptation
We introduce Margin Disparity Discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss, and to the minimax optimization for easier training.
On the Adaptivity of Stochastic Gradient-Based Optimization
Stochastic-gradient-based optimization has been a core enabling methodology in applications to large-scale problems in machine learning and related areas.
HopSkipJumpAttack: A Query-Efficient Decision-Based Attack
We develop HopSkipJumpAttack, a family of algorithms based on a novel estimate of the gradient direction using binary information at the decision boundary.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial.
Acceleration via Symplectic Discretization of High-Resolution Differential Equations
We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method.
A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm
In this note, we derive concentration inequalities for random vectors with subGaussian norm (a generalization of both subGaussian random vectors and norm bounded random vectors), which are tight up to logarithmic factors.
LS-Tree: Model Interpretation When the Data Are Linguistic
We study the problem of interpreting trained classification models in the setting of linguistic data sets.
Cost-Effective Incentive Allocation via Structured Counterfactual Inference
We address a practical problem ubiquitous in modern marketing campaigns, in which a central agent tries to learn a policy for allocating strategic financial incentives to customers and observes only bandit feedback.
Is There an Analog of Nesterov Acceleration for MCMC?
We formulate gradient-based Markov chain Monte Carlo (MCMC) sampling as optimization on the space of probability measures, with Kullback-Leibler (KL) divergence as the objective functional.
Quantitative Weak Convergence for Discrete Stochastic Processes
In this paper, we quantitative convergence in $W_2$ for a family of Langevin-like stochastic processes that includes stochastic gradient descent and related gradient-based algorithms.
What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
Minimax optimization has found extensive applications in modern machine learning, in settings such as generative adversarial networks (GANs), adversarial training and multi-agent reinforcement learning.
BIG-bench Machine Learning
Multi-agent Reinforcement Learning
+1
Sharp Analysis of Expectation-Maximization for Weakly Identifiable Models
We study a class of weakly identifiable location-scale mixture models for which the maximum likelihood estimates based on $n$ i. i. d.
Theoretically Principled Trade-off between Robustness and Accuracy
We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples.
Ranked #3 on
Adversarial Attack
on CIFAR-10
On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithms
We show that a greedy variant of the classical Sinkhorn algorithm, known as the \emph{Greenkhorn algorithm}, can be improved to $\widetilde{\mathcal{O}}(n^2\varepsilon^{-2})$, improving on the best known complexity bound of $\widetilde{\mathcal{O}}(n^2\varepsilon^{-3})$.
Data Structures and Algorithms
On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
We propose local symplectic surgery, a two-timescale procedure for finding local Nash equilibria in two-player zero-sum games.
Asynchronous Online Testing of Multiple Hypotheses
We consider the problem of asynchronous online testing, aimed at providing control of the false discovery rate (FDR) during a continual stream of data collection and testing, where each test may be a sequential test that can start and stop at arbitrary times.
Theoretical guarantees for EM under misspecified Gaussian mixture models
We provide two classes of theoretical guarantees: first, we characterize the bias introduced due to the misspecification; and second, we prove that population EM converges at a geometric rate to the model projection under a suitable initialization condition.
Generalized Zero-Shot Learning with Deep Calibration Network
A technical challenge of deep learning is recognizing target classes without seen data.
Gen-Oja: Simple & Efficient Algorithm for Streaming Generalized Eigenvector Computation
In this paper, we study the problems of principle Generalized Eigenvector computation and Canonical Correlation Analysis in the stochastic setting.
Sampling Can Be Faster Than Optimization
Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years.
Gen-Oja: A Two-time-scale approach for Streaming CCA
In this paper, we study the problems of principal Generalized Eigenvector computation and Canonical Correlation Analysis in the stochastic setting.
A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN).
Probabilistic Multilevel Clustering via Composite Transportation Distance
We propose a novel probabilistic approach to multilevel clustering problems based on composite transportation distance, which is a variant of transportation distance where the underlying metric is Kullback-Leibler divergence.
Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
We also show that these ODEs are more accurate surrogates for the underlying algorithms; in particular, they not only distinguish between NAG-SC and Polyak's heavy-ball method, but they allow the identification of a term that we refer to as "gradient correction" that is present in NAG-SC but not in the heavy-ball method and is responsible for the qualitative difference in convergence of the two methods.
Evaluating Sensitivity to the Stick-Breaking Prior in Bayesian Nonparametrics
Bayesian models based on the Dirichlet process and other stick-breaking priors have been proposed as core ingredients for clustering, topic modeling, and other unsupervised learning tasks.
Methodology
Rao-Blackwellized Stochastic Gradients for Discrete Distributions
We wish to compute the gradient of an expectation over a finite or countably infinite sample space having $K \leq \infty$ categories.
Singularity, Misspecification, and the Convergence Rate of EM
A line of recent work has analyzed the behavior of the Expectation-Maximization (EM) algorithm in the well-specified setting, in which the population likelihood is locally strongly concave around its maximizing argument.
A Deep Generative Model for Semi-Supervised Classification with Noisy Labels
Class labels are often imperfectly observed, due to mistakes and to genuine ambiguity among classes.
L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data
We study instancewise feature importance scoring as a method for model interpretation.
Is Q-learning Provably Efficient?
We prove that, in an episodic MDP setting, Q-learning with UCB exploration achieves regret $\tilde{O}(\sqrt{H^3 SAT})$, where $S$ and $A$ are the numbers of states and actions, $H$ is the number of steps per episode, and $T$ is the total number of steps.
Fundamental limits of detection in the spiked Wigner model
We study the fundamental limits of detecting the presence of an additive rank-one perturbation, or spike, to a Wigner matrix.
Improved Sample Complexity for Stochastic Compositional Variance Reduced Gradient
Convex composition optimization is an emerging topic that covers a wide range of applications arising from stochastic optimal control, reinforcement learning and multi-stage stochastic programming.
A Swiss Army Infinitesimal Jackknife
This linear approximation is sometimes known as the "infinitesimal jackknife" in the statistics literature, where it is mostly used to as a theoretical tool to prove asymptotic results.
Methodology
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data
We present a probabilistic framework for studying adversarial attacks on discrete data.
Information Constraints on Auto-Encoding Variational Bayes
We show how to apply this method to a range of problems, including the problems of learning invariant representations and the learning of interpretable representations.
Sharp convergence rates for Langevin dynamics in the nonconvex setting
We study the problem of sampling from a distribution $p^*(x) \propto \exp\left(-U(x)\right)$, where the function $U$ is $L$-smooth everywhere and $m$-strongly convex outside a ball of radius $R$, but potentially nonconvex inside this ball.
On the Local Minima of the Empirical Risk
Our objective is to find the $\epsilon$-approximate local minima of the underlying function $F$ while avoiding the shallow local minima---arising because of the tolerance $\nu$---which exist only in $f$.
On Nonlinear Dimensionality Reduction, Linear Smoothing and Autoencoding
There is limited basis for using existing NLDR theory for deriving new algorithms.
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.
Averaging Stochastic Gradient Descent on Riemannian Manifolds
We consider the minimization of a function defined on a Riemannian manifold $\mathcal{M}$ accessible only through unbiased estimates of its gradients.
Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification
We prove that the ordinary least-squares (OLS) estimator attains nearly minimax optimal performance for the identification of linear dynamical systems from a single observed trajectory.
Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
We introduce instancewise feature selection as a methodology for model interpretation.
Detection limits in the high-dimensional spiked rectangular model
This region is shaped by the prior in a non-trivial way.
On the Theory of Variance Reduction for Stochastic Gradient Monte Carlo
We provide convergence guarantees in Wasserstein distance for a variety of variance-reduction methods: SAGA Langevin diffusion, SVRG Langevin diffusion and control-variate underdamped Langevin diffusion.
RLlib: Abstractions for Distributed Reinforcement Learning
Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation.
Ray: A Distributed Framework for Emerging AI Applications
To meet the performance requirements, Ray employs a distributed scheduler and a distributed and fault-tolerant store to manage the system's control state.
Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting.
Stochastic Cubic Regularization for Fast Nonconvex Optimization
This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006].
First-order Methods Almost Always Avoid Saddle Points
We establish that first-order methods avoid saddle points for almost all initializations.
Online control of the false discovery rate with decaying memory
In the online multiple testing problem, p-values corresponding to different null hypotheses are observed one by one, and the decision of whether or not to reject the current hypothesis must be made immediately, after which the next p-value is observed.
DAGGER: A sequential algorithm for FDR control on DAGs
We propose a linear-time, single-pass, top-down algorithm for multiple testing on directed acyclic graphs (DAGs), where nodes represent hypotheses and edges specify a partial ordering in which hypotheses must be tested.
Covariances, Robustness, and Variational Bayes
The estimates for MFVB posterior covariances rely on a result from the classical Bayesian robustness literature relating derivatives of posterior expectations to posterior covariances and include the Laplace approximation as a special case.
Methodology
Partial Transfer Learning with Selective Adversarial Networks
Existing domain adversarial networks assume fully shared label space across domains.
Underdamped Langevin MCMC: A non-asymptotic analysis
We study the underdamped Langevin diffusion when the log of the target distribution is smooth and strongly concave.
Kernel Feature Selection via Conditional Covariance Minimization
We propose a method for feature selection that employs kernel-based measures of independence to find a subset of covariates that is maximally predictive of the response.
Fast Black-box Variational Inference through Stochastic Trust-Region Optimization
We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick.
Gradient Descent Can Take Exponential Time to Escape Saddle Points
Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape.
Conditional Adversarial Domain Adaptation
Adversarial learning has been embedded into deep networks to learn disentangled and transferable representations for domain adaptation.
Ranked #7 on
Domain Adaptation
on USPS-to-MNIST
A unified treatment of multiple testing with prior knowledge using the p-filter
There is a significant literature on methods for incorporating knowledge into multiple testing procedures so as to improve their power and precision.
Real-Time Machine Learning: The Missing Pieces
Machine learning applications are increasingly deployed not only to serve predictions using static models, but also as tightly-integrated components of feedback loops involving dynamic, real-time decision making.
How to Escape Saddle Points Efficiently
This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i. e., it is almost "dimension-free").
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning.
Less than a Single Pass: Stochastically Controlled Stochastic Gradient Method
We develop and analyze a procedure for gradient-based optimization that we refer to as stochastically controlled stochastic gradient (SCSG).
CYCLADES: Conflict-free Asynchronous Machine Learning
We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting.
Communication-Efficient Distributed Statistical Inference
CSL provides a communication-efficient surrogate to the global likelihood that can be used for low-dimensional estimation, high-dimensional regularized estimation and Bayesian inference.
Deep Transfer Learning with Joint Adaptation Networks
Deep networks have been successfully applied to learn transferable features for adapting models from a source domain to a different target domain.
Ranked #2 on
Domain Adaptation
on HMDBfull-to-UCF
Multi-Source Unsupervised Domain Adaptation
Transfer Learning
Function-Specific Mixing Times and Concentration Away from Equilibrium
These results show that it is possible for empirical expectations of functions to concentrate long before the underlying chain has mixed in the classical sense, and we show that the concentration rates we achieve are optimal up to constants.
On kernel methods for covariates that are rankings
This paper studies the use of reproducing kernel Hilbert space methods for learning from permutation-valued features.
A Variational Perspective on Accelerated Methods in Optimization
We show that there is a Lagrangian functional that we call the \emph{Bregman Lagrangian} which generates a large class of accelerated methods in continuous time, including (but not limited to) accelerated gradient descent, its non-Euclidean extension, and accelerated higher-order gradient methods.
Asymptotic behavior of $\ell_p$-based Laplacian regularization in semi-supervised learning
Together, these properties show that $p = d+1$ is an optimal choice, yielding a function estimate $\hat{f}$ that is both smooth and non-degenerate, while remaining maximally sensitive to $P$.
Gradient Descent Converges to Minimizers
We show that gradient descent converges to a local minimizer, almost surely with random initialization.
Unsupervised Domain Adaptation with Residual Transfer Networks
In this paper, we propose a new approach to domain adaptation in deep networks that can jointly learn adaptive classifiers and transferable features from labeled data in the source domain and unlabeled data in the target domain.
A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation
We derive a new discrepancy statistic for measuring differences between two probability distributions based on combining Stein's identity with the reproducing kernel Hilbert space theory.
L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
Despite the importance of sparsity in many large-scale applications, there are few methods for distributed optimization of sparsity-inducing objectives.
Distributed Optimization with Arbitrary Local Solvers
To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally, and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods.
Learning Halfspaces and Neural Networks with Random Initialization
For loss functions that are $L$-Lipschitz continuous, we present algorithms to learn halfspaces and multi-layer neural networks that achieve arbitrarily small excess risk $\epsilon>0$.
SparkNet: Training Deep Networks in Spark
We introduce SparkNet, a framework for training deep networks in Spark.
$\ell_1$-regularized Neural Networks are Improperly Learnable in Polynomial Time
The sample complexity and the time complexity of the presented method are polynomial in the input dimension and in $(1/\epsilon,\log(1/\delta), F(k, L))$, where $F(k, L)$ is a function depending on $(k, L)$ and on the activation function, independent of the number of neurons.
A Linearly-Convergent Stochastic L-BFGS Algorithm
We propose a new stochastic L-BFGS algorithm and prove a linear convergence rate for strongly convex and smooth functions.
Perturbed Iterate Analysis for Asynchronous Stochastic Optimization
We demonstrate experimentally on a 16-core machine that the sparse and parallel version of SVRG is in some cases more than four orders of magnitude faster than the standard SVRG algorithm.
Parallel Correlation Clustering on Big Graphs
We present C4 and ClusterWild!, two algorithms for parallel correlation clustering that run in a polylogarithmic number of rounds and achieve nearly linear speedups, provably.
Splash: User-friendly Programming Interface for Parallelizing Stochastic Algorithms
Splash consists of a programming interface and an execution engine.
On the accuracy of self-normalized log-linear models
Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces.
Variational consensus Monte Carlo
Practitioners of Bayesian statistics have long depended on Markov chain Monte Carlo (MCMC) to obtain samples from intractable posterior distributions.
On the Computational Complexity of High-Dimensional Bayesian Variable Selection
We study the computational complexity of Markov chain Monte Carlo (MCMC) methods for high-dimensional Bayesian linear regression under sparsity constraints.
Optimal prediction for sparse linear models? Lower bounds for coordinate-separable M-estimators
In this paper, we show that the slow rate is intrinsic to a broad class of M-estimators.
Trust Region Policy Optimization
We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement.
Adding vs. Averaging in Distributed Primal-Dual Optimization
Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck.
Learning Transferable Features with Deep Adaptation Networks
Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks for domain adaptation.
Ranked #3 on
Domain Adaptation
on Synth Digits-to-SVHN
A General Analysis of the Convergence of ADMM
We provide a new proof of the linear convergence of the alternating direction method of multipliers (ADMM) when one of the objective terms is strongly convex.
Optimization and Control Numerical Analysis
Distributed Estimation of Generalized Matrix Rank: Efficient Algorithms and Lower Bounds
We study the following generalized matrix rank estimation problem: given an $n \times n$ matrix and a constant $c \geq 0$, estimate the number of eigenvalues that are greater than $c$.
TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries
The proliferation of massive datasets combined with the development of sophisticated analytical techniques have enabled a wide variety of novel applications such as improved product recommendations, automatic image tagging, and improved speech-driven interfaces.
Parallel Double Greedy Submodular Maximization
Many machine learning problems can be reduced to the maximization of submodular functions.
The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
In this work, we present Velox, a new component of the Berkeley Data Analytics Stack.
Databases
Communication-Efficient Distributed Dual Coordinate Ascent
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning.
On the Convergence Rate of Decomposable Submodular Function Minimization
Submodular functions describe a variety of discrete problems in machine learning, signal processing, and computer vision.
Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing
Crowdsourcing is a popular paradigm for effectively collecting labels at low cost.
Optimality guarantees for distributed statistical estimation
Large data sets often require performing distributed statistical estimation, with a full data set split across multiple machines and limited communication between machines.
Particle Gibbs with Ancestor Sampling
Particle Markov chain Monte Carlo (PMCMC) is a systematic way of combining the two main tools used for Monte Carlo statistical inference: sequential Monte Carlo (SMC) and Markov chain Monte Carlo (MCMC).
Optimal rates for zero-order convex optimization: the power of two function evaluations
We consider derivative-free algorithms for stochastic and non-stochastic convex optimization problems that use only function values rather than gradients.
Information-theoretic lower bounds for distributed statistical estimation with communication constraints
We establish minimax risk lower bounds for distributed statistical estimation given a budget $B$ of the total number of bits that may be communicated.
A Comparative Framework for Preconditioned Lasso Algorithms
The Lasso is a cornerstone of modern multivariate data analysis, yet its performance suffers in the common situation in which covariates are correlated.
Local Privacy and Minimax Bounds: Sharp Rates for Probability Estimation
We provide a detailed study of the estimation of probability distributions---discrete and continuous---in a stringent setting in which data is kept private even from the statistician.
Estimation, Optimization, and Parallelism when Data is Sparse
We study stochastic optimization problems when the \emph{data} is sparse, which is in a sense dual to the current understanding of high-dimensional statistical learning and optimization.
MLI: An API for Distributed Machine Learning
MLI is an Application Programming Interface designed to address the challenges of building Machine Learn- ing algorithms in a distributed setting based on data-centric computing.
On statistics, computation and scalability
How should statistical procedures be designed so as to be scalable computationally to the massive datasets that are increasingly the norm?
Mixed Membership Models for Time Series
Although much of the literature on mixed membership models considers the setting in which exchangeable collections of data are associated with each member of a set of entities, it is equally natural to consider problems in which an entire time series is viewed as an entity and the goal is to characterize the time series in terms of a set of underlying dynamic attributes or "dynamic regimes".
Joint modeling of multiple time series via the beta process with application to motion capture segmentation
We propose a Bayesian nonparametric approach to the problem of jointly modeling multiple related time series.
Optimistic Concurrency Control for Distributed Unsupervised Learning
Research on distributed machine learning algorithms has focused primarily on one of two extremes - algorithms that obey strict concurrency constraints or algorithms that obey few or no such constraints.
Streaming Variational Bayes
We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior.
Distributed Low-rank Subspace Segmentation
Vision problems ranging from image clustering to motion segmentation to semi-supervised learning can naturally be framed as subspace segmentation problems, in which one aims to recover multiple low-dimensional subspaces from noisy and corrupted input data.
Loopy Belief Propagation for Approximate Inference: An Empirical Study
Recently, researchers have demonstrated that loopy belief propagation - the use of Pearls polytree algorithm IN a Bayesian network WITH loops OF error- correcting codes. The most dramatic instance OF this IS the near Shannon - limit performance OF Turbo Codes codes whose decoding algorithm IS equivalent TO loopy belief propagation IN a chain - structured Bayesian network.
Variational MCMC
One of these algorithms is a mixture of two MCMC kernels: a random walk Metropolis kernel and a blockMetropolis-Hastings (MH) kernel with a variational approximation as proposaldistribution.
Small-Variance Asymptotics for Exponential Family Dirichlet Process Mixture Models
Links between probabilistic and non-probabilistic learning algorithms can arise by performing small-variance asymptotics, i. e., letting the variance of particular distributions in a graphical model go to zero.
Ancestor Sampling for Particle Gibbs
We present a novel method in the family of particle MCMC methods that we refer to as particle Gibbs with ancestor sampling (PG-AS).
Finite Sample Convergence Rates of Zero-Order Stochastic Optimization Methods
We consider derivative-free algorithms for stochastic optimization problems that use only noisy function values rather than gradients, analyzing their finite-sample convergence rates.
Nested Hierarchical Dirichlet Processes
We develop a nested hierarchical Dirichlet process (nHDP) for hierarchical topic modeling.
Privacy Aware Learning
We study statistical risk minimization problems under a privacy model in which the data is kept confidential even from the learner.
Active Learning for Crowd-Sourced Databases
Based on this observation, we present two new active learning algorithms to combine humans and algorithms together in a crowd-sourced database.
The asymptotics of ranking algorithms
With these negative results as motivation, we present a new approach to supervised ranking based on aggregation of partial preferences, and we develop $U$-statistic-based empirical risk minimization procedures.
Bayesian Bias Mitigation for Crowdsourcing
This approach can account for more complex bias patterns that arise in ambiguous or hard labeling tasks and allows us to merge data curation and learning into a single computation.
Divide-and-Conquer Matrix Factorization
This work introduces Divide-Factor-Combine (DFC), a parallel divide-and-conquer framework for noisy matrix factorization.
Combinatorial clustering and the beta negative binomial process
We show that the NBP is conjugate to the beta process, and we characterize the posterior distribution under the beta-negative binomial process (BNBP) and hierarchical models based on the BNBP (the HBNBP).
Distributed Matrix Completion and Robust Factorization
If learning methods are to scale to the massive sizes of modern datasets, it is essential for the field of machine learning to embrace parallel and distributed computing.
Random Conic Pursuit for Semidefinite Programming
We present a novel algorithm, Random Conic Pursuit, that solves semidefinite programs (SDPs) via repeated optimization over randomly selected two-dimensional subcones of the PSD cone.
Variational Inference over Combinatorial Spaces
Since the discovery of sophisticated fully polynomial randomized algorithms for a range of #P problems (Karzanov et al., 1991; Jerrum et al., 2001; Wilson, 2004), theoretical work on approximate inference in combinatorial spaces has focused on Markov chain Monte Carlo methods.