Search Results for author: Dan Garber

Found 36 papers, 0 papers with code

Online Convex Optimization in the Random Order Model

no code implementations ICML 2020 Dan Garber, Gal Korcia, Kfir Levy

Focusing on two important families of online tasks, one which generalizes online linear and logistic regression, and the other being online PCA, we show that under standard well-conditioned-data assumptions (that are often being made in the corresponding offline settings), standard online gradient descent (OGD) methods become much more efficient in the random-order model.

regression

Projection-Free Online Convex Optimization with Time-Varying Constraints

no code implementations13 Feb 2024 Dan Garber, Ben Kretzu

We also present a more efficient algorithm that requires only first-order oracle access to the soft constraints and achieves similar bounds w. r. t.

From Oja's Algorithm to the Multiplicative Weights Update Method with Applications

no code implementations24 Oct 2023 Dan Garber

Oja's algorithm is a well known online algorithm studied mainly in the context of stochastic principal component analysis.

Efficiency of First-Order Methods for Low-Rank Tensor Recovery with the Tensor Nuclear Norm Under Strict Complementarity

no code implementations3 Aug 2023 Dan Garber, Atara Kaplan

For a smooth objective function, when initialized in certain proximity of an optimal solution which satisfies SC, standard projected gradient methods only require SVD computations (for projecting onto the tensor nuclear norm ball) of rank that matches the tubal rank of the optimal solution.

Projection-free Online Exp-concave Optimization

no code implementations9 Feb 2023 Dan Garber, Ben Kretzu

We consider the setting of online convex optimization (OCO) with \textit{exp-concave} losses.

Faster Projection-Free Augmented Lagrangian Methods via Weak Proximal Oracle

no code implementations25 Oct 2022 Dan Garber, Tsur Livney, Shoham Sabach

This paper considers a convex composite optimization problem with affine constraints, which includes problems that take the form of minimizing a smooth convex objective function over the intersection of (simple) convex sets, or regularized with multiple (simple) functions.

Low-Rank Mirror-Prox for Nonsmooth and Low-Rank Matrix Optimization Problems

no code implementations23 Jun 2022 Dan Garber, Atara Kaplan

Low-rank and nonsmooth matrix optimization problems capture many fundamental tasks in statistics and machine learning.

Frank-Wolfe-based Algorithms for Approximating Tyler's M-estimator

no code implementations19 Jun 2022 Lior Danon, Dan Garber

All three variants are shown to provably converge to the optimal solution with sublinear rate, under standard assumptions, despite the fact that the underlying optimization problem is not convex nor smooth.

New Projection-free Algorithms for Online Convex Optimization with Adaptive Regret Guarantees

no code implementations9 Feb 2022 Dan Garber, Ben Kretzu

Concretely, when assuming the availability of a linear optimization oracle (LOO) for the feasible set, on a sequence of length $T$, our algorithms guarantee $O(T^{3/4})$ adaptive regret and $O(T^{3/4})$ adaptive expected regret, for the full-information and bandit settings, respectively, using only $O(T)$ calls to the LOO.

Local Linear Convergence of Gradient Methods for Subspace Optimization via Strict Complementarity

no code implementations8 Feb 2022 Dan Garber, Ron Fisher

We consider optimization problems in which the goal is find a $k$-dimensional subspace of $\mathbb{R}^n$, $k<<n$, which minimizes a convex and smooth loss.

Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems

no code implementations NeurIPS 2021 Dan Garber, Atara Kaplan

Low-rank and nonsmooth matrix optimization problems capture many fundamental tasks in statistics and machine learning.

Frank-Wolfe with a Nearest Extreme Point Oracle

no code implementations3 Feb 2021 Dan Garber, Noam Wolf

We consider variants of the classical Frank-Wolfe algorithm for constrained smooth convex minimization, that instead of access to the standard oracle for minimizing a linear function over the feasible set, have access to an oracle that can find an extreme point of the feasible set that is closest in Euclidean distance to a given vector.

On the Efficient Implementation of the Matrix Exponentiated Gradient Algorithm for Low-Rank Matrix Optimization

no code implementations18 Dec 2020 Dan Garber, Atara Kaplan

In this work we propose an efficient implementations of MEG, both with deterministic and stochastic gradients, which are tailored for optimization with low-rank matrices, and only use a single low-rank SVD computation on each iteration.

Revisiting Projection-free Online Learning: the Strongly Convex Case

no code implementations15 Oct 2020 Dan Garber, Ben Kretzu

We also revisit the bandit setting under strong convexity and prove a similar bound of $\tilde O(T^{2/3})$ (instead of $O(T^{3/4})$ without strong convexity).

Revisiting Frank-Wolfe for Polytopes: Strict Complementarity and Sparsity

no code implementations NeurIPS 2020 Dan Garber

In recent years it was proved that simple modifications of the classical Frank-Wolfe algorithm (aka conditional gradient algorithm) for smooth convex minimization over convex and compact polytopes, converge with linear rate, assuming the objective function has the quadratic growth property.

On the Convergence of Stochastic Gradient Descent with Low-Rank Projections for Convex Low-Rank Matrix Problems

no code implementations31 Jan 2020 Dan Garber

Our main result shows that under this condition which involves the eigenvalues of the gradient vector at optimal points, SGD with mini-batches, when initialized with a "warm-start" point, produces iterates that are low-rank with high probability, and hence only a low-rank SVD computation is required on each iteration.

Matrix Completion Retrieval

Linear Convergence of Frank-Wolfe for Rank-One Matrix Recovery Without Strong Convexity

no code implementations3 Dec 2019 Dan Garber

We consider convex optimization problems which are widely used as convex relaxations for low-rank matrix recovery problems.

Retrieval

Improved Regret Bounds for Projection-free Bandit Convex Optimization

no code implementations8 Oct 2019 Dan Garber, Ben Kretzu

We revisit the challenge of designing online algorithms for the bandit convex optimization problem (BCO) which are also scalable to high dimensional problems.

On the Convergence of Projected-Gradient Methods with Low-Rank Projections for Smooth Convex Minimization over Trace-Norm Balls and Related Problems

no code implementations5 Feb 2019 Dan Garber

We also quantify the effect of "over-parameterization", i. e., using SVD computations with higher rank, on the radius of this ball, showing it can increase dramatically with moderately larger rank.

On the Regret Minimization of Nonconvex Online Gradient Ascent for Online PCA

no code implementations27 Sep 2018 Dan Garber

In this paper we focus on the problem of Online Principal Component Analysis in the regret minimization framework.

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

no code implementations27 Sep 2018 Dan Garber, Atara Kaplan

However, such problems are highly challenging to solve in large-scale: the low-rank promoting term prohibits efficient implementations of proximal methods for composite optimization and even simple subgradient methods.

Stochastic Optimization

Learning of Optimal Forecast Aggregation in Partial Evidence Environments

no code implementations20 Feb 2018 Yakov Babichenko, Dan Garber

We focus on the question whether the aggregator can learn to aggregate optimally the forecasts of the experts, where the optimal aggregation is the Bayesian aggregation that takes into account all the information (evidence) in the system.

Improved Complexities of Conditional Gradient-Type Methods with Applications to Robust Matrix Recovery Problems

no code implementations15 Feb 2018 Dan Garber, Shoham Sabach, Atara Kaplan

Motivated by robust matrix recovery problems such as Robust Principal Component Analysis, we consider a general optimization problem of minimizing a smooth and strongly convex loss function applied to the sum of two blocks of variables, where each block of variables is constrained or regularized individually.

Logarithmic Regret for Online Gradient Descent Beyond Strong Convexity

no code implementations13 Feb 2018 Dan Garber

In particular, our results hold for \textit{semi-adversarial} settings in which the data is a combination of an arbitrary (adversarial) sequence and a stochastic sequence, which might provide reasonable approximation for many real-world sequences, or under a natural assumption that the data is low-rank.

Efficient Online Linear Optimization with Approximation Algorithms

no code implementations NeurIPS 2017 Dan Garber

This setting is in particular interesting since it captures natural online extensions of well-studied \textit{offline} linear optimization problems which are NP-hard, yet admit efficient approximation algorithms.

Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis

no code implementations ICML 2017 Dan Garber, Ohad Shamir, Nathan Srebro

We study algorithms for estimating the leading principal component of the population covariance matrix that are both communication-efficient and achieve estimation error of the order of the centralized ERM solution that uses all $mn$ samples.

Efficient coordinate-wise leading eigenvector computation

no code implementations25 Feb 2017 Jialei Wang, Weiran Wang, Dan Garber, Nathan Srebro

We develop and analyze efficient "coordinate-wise" methods for finding the leading eigenvector, where each step involves only a vector-vector product.

regression

Stochastic Canonical Correlation Analysis

no code implementations21 Feb 2017 Chao Gao, Dan Garber, Nathan Srebro, Jialei Wang, Weiran Wang

We study the sample complexity of canonical correlation analysis (CCA), \ie, the number of samples needed to estimate the population canonical correlation and directions up to arbitrarily small error.

Stochastic Optimization

Faster Eigenvector Computation via Shift-and-Invert Preconditioning

no code implementations26 May 2016 Dan Garber, Elad Hazan, Chi Jin, Sham M. Kakade, Cameron Musco, Praneeth Netrapalli, Aaron Sidford

We give faster algorithms and improved sample complexities for estimating the top eigenvector of a matrix $\Sigma$ -- i. e. computing a unit vector $x$ such that $x^T \Sigma x \ge (1-\epsilon)\lambda_1(\Sigma)$: Offline Eigenvector Estimation: Given an explicit $A \in \mathbb{R}^{n \times d}$ with $\Sigma = A^TA$, we show how to compute an $\epsilon$ approximate top eigenvector in time $\tilde O([nnz(A) + \frac{d*sr(A)}{gap^2} ]* \log 1/\epsilon )$ and $\tilde O([\frac{nnz(A)^{3/4} (d*sr(A))^{1/4}}{\sqrt{gap}} ] * \log 1/\epsilon )$.

Stochastic Optimization

Linear-memory and Decomposition-invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes

no code implementations NeurIPS 2016 Dan Garber, Ofer Meshi

Moreover, in case the optimal solution is sparse, the new convergence rate replaces a factor which is at least linear in the dimension in previous works, with a linear dependence on the number of non-zeros in the optimal solution.

Structured Prediction

Faster Projection-free Convex Optimization over the Spectrahedron

no code implementations NeurIPS 2016 Dan Garber

Minimizing a convex function over the spectrahedron, i. e., the set of all positive semidefinite matrices with unit trace, is an important optimization task with many applications in optimization, machine learning, and signal processing.

Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis

no code implementations NeurIPS 2016 Weiran Wang, Jialei Wang, Dan Garber, Nathan Srebro

We study the stochastic optimization of canonical correlation analysis (CCA), whose objective is nonconvex and does not decouple over training samples.

Stochastic Optimization

Fast and Simple PCA via Convex Optimization

no code implementations18 Sep 2015 Dan Garber, Elad Hazan

The problem of principle component analysis (PCA) is traditionally solved by spectral or algebraic methods.

Faster Rates for the Frank-Wolfe Method over Strongly-Convex Sets

no code implementations5 Jun 2014 Dan Garber, Elad Hazan

In this paper we consider the special case of optimization over strongly convex sets, for which we prove that the vanila FW method converges at a rate of $\frac{1}{t^2}$.

Approximating Semidefinite Programs in Sublinear Time

no code implementations NeurIPS 2011 Dan Garber, Elad Hazan

In recent years semidefinite optimization has become a tool of major importance in various optimization and machine learning problems.

Cannot find the paper you are looking for? You can Submit a new open access paper.