Search Results for author: Raman Arora

Found 43 papers, 10 papers with code

Communication-Efficient Federated Learning with Sketching

no code implementations ICML 2020 Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Vladimir Braverman, Joseph Gonzalez, Ion Stoica, Raman Arora

A key insight in the design of FedSketchedSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch.

Federated Learning

Differentially Private Generalized Linear Models Revisited

no code implementations6 May 2022 Raman Arora, Raef Bassily, Cristóbal Guzmán, Michael Menart, Enayat Ullah

For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) $\Theta\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^*\Vert}{\sqrt{n\epsilon}},\frac{\sqrt{\text{rank}}\Vert w^*\Vert}{n\epsilon}\right\}\right)$, where $\text{rank}$ is the rank of the design matrix.

Model Selection

A Risk-Sensitive Policy Gradient Method

no code implementations29 Sep 2021 Jared Markowitz, Ryan Gardner, Ashley Llorens, Raman Arora, I-Jeng Wang

Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating a policy.

Decision Making

Machine Unlearning via Algorithmic Stability

no code implementations25 Feb 2021 Enayat Ullah, Tung Mai, Anup Rao, Ryan Rossi, Raman Arora

Our key contribution is the design of corresponding efficient unlearning algorithms, which are based on constructing a (maximal) coupling of Markov chains for the noisy SGD procedure.

On Convergence and Generalization of Dropout Training

no code implementations NeurIPS 2020 Poorya Mianjy, Raman Arora

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations.

FetchSGD: Communication-Efficient Federated Learning with Sketching

no code implementations15 Jul 2020 Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, Raman Arora

A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch.

Federated Learning

Is Network the Bottleneck of Distributed Training?

1 code implementation17 Jun 2020 Zhen Zhang, Chaokun Chang, Haibin Lin, Yida Wang, Raman Arora, Xin Jin

As such, we advocate that the real challenge of distributed training is for the network community to develop high-performance network transport to fully utilize the network capacity and achieve linear scale-out.

Corralling Stochastic Bandit Algorithms

no code implementations16 Jun 2020 Raman Arora, Teodor V. Marinov, Mehryar Mohri

We study the problem of corralling stochastic bandit algorithms, that is combining multiple bandit algorithms designed for a stochastic environment, with the goal of devising a corralling algorithm that performs almost as well as the best base algorithm.

Dropout: Explicit Forms and Capacity Control

no code implementations ICLR 2020 Raman Arora, Peter Bartlett, Poorya Mianjy, Nathan Srebro

In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks.

Matrix Completion

Multiview Representation Learning for a Union of Subspaces

no code implementations30 Dec 2019 Nils Holzenberger, Raman Arora

Canonical correlation analysis (CCA) is a popular technique for learning representations that are maximally correlated across multiple views in data.

Representation Learning

Efficient Convex Relaxations for Streaming PCA

1 code implementation NeurIPS 2019 Raman Arora, Teodor Vanislavov Marinov

We revisit two algorithms, matrix stochastic gradient (MSG) and $\ell_2$-regularized MSG (RMSG), that are instances of stochastic gradient descent (SGD) on a convex relaxation to principal component analysis (PCA).

Bandits with Feedback Graphs and Switching Costs

no code implementations NeurIPS 2019 Raman Arora, Teodor V. Marinov, Mehryar Mohri

We give a new algorithm whose regret guarantee depends only on the domination number of the graph.

On Dropout and Nuclear Norm Regularization

1 code implementation28 May 2019 Poorya Mianjy, Raman Arora

We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss.

Communication-efficient distributed SGD with Sketching

2 code implementations NeurIPS 2019 Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Vladimir Braverman, Ion Stoica, Raman Arora

Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time.

Streaming Kernel PCA with \tilde{O}(\sqrt{n}) Random Features

no code implementations NeurIPS 2018 Md Enayat Ullah, Poorya Mianjy, Teodor Vanislavov Marinov, Raman Arora

We study the statistical and computational aspects of kernel principal component analysis using random Fourier features and show that under mild assumptions, $O(\sqrt{n} \log n)$ features suffices to achieve $O(1/\epsilon^2)$ sample complexity.

Differentially Private Robust Low-Rank Approximation

no code implementations NeurIPS 2018 Raman Arora, Vladimir Braverman, Jalaj Upadhyay

In this paper, we study the following robust low-rank matrix approximation problem: given a matrix $A \in \R^{n \times d}$, find a rank-$k$ matrix $B$, while satisfying differential privacy, such that $ \norm{ A - B }_p \leq \alpha \mathsf{OPT}_k(A) + \tau,$ where $\norm{ M }_p$ is the entry-wise $\ell_p$-norm and $\mathsf{OPT}_k(A):=\min_{\mathsf{rank}(X) \leq k} \norm{ A - X}_p$.

Learning from Multiview Correlations in Open-Domain Videos

no code implementations21 Nov 2018 Nils Holzenberger, Shruti Palaskar, Pranava Madhyastha, Florian Metze, Raman Arora

This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.

Representation Learning

Policy Regret in Repeated Games

no code implementations NeurIPS 2018 Raman Arora, Michael Dinitz, Teodor V. Marinov, Mehryar Mohri

We revisit the notion of policy regret and first show that there are online learning settings in which policy regret and external regret are incompatible: any sequence of play that achieves a favorable regret with respect to one definition must do poorly with respect to the other.

online learning

Streaming Kernel PCA with $\tilde{O}(\sqrt{n})$ Random Features

1 code implementation2 Aug 2018 Enayat Ullah, Poorya Mianjy, Teodor V. Marinov, Raman Arora

We study the statistical and computational aspects of kernel principal component analysis using random Fourier features and show that under mild assumptions, $O(\sqrt{n} \log n)$ features suffices to achieve $O(1/\epsilon^2)$ sample complexity.

Stochastic PCA with $\ell_2$ and $\ell_1$ Regularization

no code implementations ICML 2018 Poorya Mianjy, Raman Arora

We revisit convex relaxation based methods for stochastic optimization of principal component analysis (PCA).

Stochastic Optimization

On the Implicit Bias of Dropout

no code implementations ICML 2018 Poorya Mianjy, Raman Arora, Rene Vidal

Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings.

Visual Robot Task Planning

1 code implementation30 Mar 2018 Chris Paxton, Yotam Barnoy, Kapil Katyal, Raman Arora, Gregory D. Hager

In this work, we propose a neural network architecture and associated planning algorithm that (1) learns a representation of the world useful for generating prospective futures after the application of high-level actions, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) uses this same representation to evaluate these actions and perform tree search to find a sequence of high-level actions in a new environment.

Imitation Learning Robot Task Planning

Learning to Imagine Manipulation Goals for Robot Task Planning

no code implementations8 Nov 2017 Chris Paxton, Kapil Katyal, Christian Rupprecht, Raman Arora, Gregory D. Hager

Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment.

Robot Task Planning

Stochastic Approximation for Canonical Correlation Analysis

no code implementations NeurIPS 2017 Raman Arora, Teodor V. Marinov, Poorya Mianjy, Nathan Srebro

We propose novel first-order stochastic approximation algorithms for canonical correlation analysis (CCA).

Deep Generalized Canonical Correlation Analysis

3 code implementations WS 2019 Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, Raman Arora

We present Deep Generalized Canonical Correlation Analysis (DGCCA) -- a method for learning nonlinear transformations of arbitrarily many views of data, such that the resulting transformations are maximally informative of each other.

Representation Learning Stochastic Optimization

Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization

no code implementations29 Dec 2016 Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, Tuo Zhao

We propose a general theory for studying the \xl{landscape} of nonconvex \xl{optimization} with underlying symmetric structures \tz{for a class of machine learning problems (e. g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}.

Understanding Deep Neural Networks with Rectified Linear Units

no code implementations ICLR 2018 Raman Arora, Amitabh Basu, Poorya Mianjy, Anirbit Mukherjee

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU).

On Faster Convergence of Cyclic Block Coordinate Descent-type Methods for Strongly Convex Minimization

no code implementations10 Jul 2016 Xingguo Li, Tuo Zhao, Raman Arora, Han Liu, Mingyi Hong

In particular, we first show that for a family of quadratic minimization problems, the iteration complexity $\mathcal{O}(\log^2(p)\cdot\log(1/\epsilon))$ of the CBCD-type methods matches that of the GD methods in term of dependency on $p$, up to a $\log^2 p$ factor.

Disease Trajectory Maps

no code implementations NeurIPS 2016 Peter Schulam, Raman Arora

To answer these questions, we propose the Disease Trajectory Map (DTM), a novel probabilistic model that learns low-dimensional representations of sparse and irregularly sampled time series.

Time Series Variational Inference

On Fast Convergence of Proximal Algorithms for SQRT-Lasso Optimization: Don't Worry About Its Nonsmooth Loss Function

no code implementations25 May 2016 Xingguo Li, Haoming Jiang, Jarvis Haupt, Raman Arora, Han Liu, Mingyi Hong, Tuo Zhao

Many machine learning techniques sacrifice convenient computational structures to gain estimation robustness and modeling flexibility.

Nonconvex Sparse Learning via Stochastic Optimization with Progressive Variance Reduction

no code implementations9 May 2016 Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, Tuo Zhao

We propose a stochastic variance reduced optimization algorithm for solving sparse learning problems with cardinality constraints.

Sparse Learning Stochastic Optimization

Embedding Lexical Features via Low-Rank Tensors

1 code implementation NAACL 2016 Mo Yu, Mark Dredze, Raman Arora, Matthew Gormley

Modern NLP models rely heavily on engineered features, which often combine word and contextual information into complex lexical features.

Relation Extraction

On Deep Multi-View Representation Learning: Objectives and Optimization

1 code implementation2 Feb 2016 Weiran Wang, Raman Arora, Karen Livescu, Jeff Bilmes

We consider learning representations (features) in the setting in which we have access to multiple unlabeled views of the data for learning while only one view is available for downstream tasks.

Representation Learning Stochastic Optimization

Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations

no code implementations7 Oct 2015 Weiran Wang, Raman Arora, Karen Livescu, Nathan Srebro

Deep CCA is a recently proposed deep neural network extension to the traditional canonical correlation analysis (CCA), and has been successful for multi-view representation learning in several domains.

Representation Learning Stochastic Optimization

Accelerated Mini-batch Randomized Block Coordinate Descent Method

no code implementations NeurIPS 2014 Tuo Zhao, Mo Yu, Yiming Wang, Raman Arora, Han Liu

When the regularization function is block separable, we can solve the minimization problems in a randomized block coordinate descent (RBCD) manner.

Sparse Learning Stochastic Optimization

Stochastic Optimization of PCA with Capped MSG

no code implementations NeurIPS 2013 Raman Arora, Andrew Cotter, Nathan Srebro

We study PCA as a stochastic optimization problem and propose a novel stochastic approximation algorithm which we refer to as "Matrix Stochastic Gradient" (MSG), as well as a practical variant, Capped MSG.

Stochastic Optimization

On Learning Rotations

no code implementations NeurIPS 2009 Raman Arora

An algorithm is presented for online learning of rotations.

online learning

Cannot find the paper you are looking for? You can Submit a new open access paper.