Search Results for author: Raman Arora

Found 54 papers, 12 papers with code

On Learning Rotations

no code implementations NeurIPS 2009 Raman Arora

An algorithm is presented for online learning of rotations.

Stochastic Optimization of PCA with Capped MSG

no code implementations NeurIPS 2013 Raman Arora, Andrew Cotter, Nathan Srebro

We study PCA as a stochastic optimization problem and propose a novel stochastic approximation algorithm which we refer to as "Matrix Stochastic Gradient" (MSG), as well as a practical variant, Capped MSG.

Stochastic Optimization

Accelerated Mini-batch Randomized Block Coordinate Descent Method

no code implementations NeurIPS 2014 Tuo Zhao, Mo Yu, Yiming Wang, Raman Arora, Han Liu

When the regularization function is block separable, we can solve the minimization problems in a randomized block coordinate descent (RBCD) manner.

Sparse Learning Stochastic Optimization

Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations

no code implementations7 Oct 2015 Weiran Wang, Raman Arora, Karen Livescu, Nathan Srebro

Deep CCA is a recently proposed deep neural network extension to the traditional canonical correlation analysis (CCA), and has been successful for multi-view representation learning in several domains.

Representation Learning Stochastic Optimization

On Deep Multi-View Representation Learning: Objectives and Optimization

1 code implementation2 Feb 2016 Weiran Wang, Raman Arora, Karen Livescu, Jeff Bilmes

We consider learning representations (features) in the setting in which we have access to multiple unlabeled views of the data for learning while only one view is available for downstream tasks.

Representation Learning Stochastic Optimization

Embedding Lexical Features via Low-Rank Tensors

1 code implementation NAACL 2016 Mo Yu, Mark Dredze, Raman Arora, Matthew Gormley

Modern NLP models rely heavily on engineered features, which often combine word and contextual information into complex lexical features.

Relation Extraction

Nonconvex Sparse Learning via Stochastic Optimization with Progressive Variance Reduction

no code implementations9 May 2016 Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, Tuo Zhao

We propose a stochastic variance reduced optimization algorithm for solving sparse learning problems with cardinality constraints.

Sparse Learning Stochastic Optimization

On Fast Convergence of Proximal Algorithms for SQRT-Lasso Optimization: Don't Worry About Its Nonsmooth Loss Function

no code implementations25 May 2016 Xingguo Li, Haoming Jiang, Jarvis Haupt, Raman Arora, Han Liu, Mingyi Hong, Tuo Zhao

Many machine learning techniques sacrifice convenient computational structures to gain estimation robustness and modeling flexibility.

regression

Disease Trajectory Maps

no code implementations NeurIPS 2016 Peter Schulam, Raman Arora

To answer these questions, we propose the Disease Trajectory Map (DTM), a novel probabilistic model that learns low-dimensional representations of sparse and irregularly sampled time series.

Time Series Time Series Analysis +1

On Faster Convergence of Cyclic Block Coordinate Descent-type Methods for Strongly Convex Minimization

no code implementations10 Jul 2016 Xingguo Li, Tuo Zhao, Raman Arora, Han Liu, Mingyi Hong

In particular, we first show that for a family of quadratic minimization problems, the iteration complexity $\mathcal{O}(\log^2(p)\cdot\log(1/\epsilon))$ of the CBCD-type methods matches that of the GD methods in term of dependency on $p$, up to a $\log^2 p$ factor.

regression

Understanding Deep Neural Networks with Rectified Linear Units

no code implementations ICLR 2018 Raman Arora, Amitabh Basu, Poorya Mianjy, Anirbit Mukherjee

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU).

Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization

no code implementations29 Dec 2016 Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, Tuo Zhao

We propose a general theory for studying the \xl{landscape} of nonconvex \xl{optimization} with underlying symmetric structures \tz{for a class of machine learning problems (e. g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}.

Retrieval

Deep Generalized Canonical Correlation Analysis

3 code implementations WS 2019 Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, Raman Arora

We present Deep Generalized Canonical Correlation Analysis (DGCCA) -- a method for learning nonlinear transformations of arbitrarily many views of data, such that the resulting transformations are maximally informative of each other.

Representation Learning Stochastic Optimization

Stochastic Approximation for Canonical Correlation Analysis

no code implementations NeurIPS 2017 Raman Arora, Teodor V. Marinov, Poorya Mianjy, Nathan Srebro

We propose novel first-order stochastic approximation algorithms for canonical correlation analysis (CCA).

Learning to Imagine Manipulation Goals for Robot Task Planning

no code implementations8 Nov 2017 Chris Paxton, Kapil Katyal, Christian Rupprecht, Raman Arora, Gregory D. Hager

Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment.

Robot Task Planning

Visual Robot Task Planning

1 code implementation30 Mar 2018 Chris Paxton, Yotam Barnoy, Kapil Katyal, Raman Arora, Gregory D. Hager

In this work, we propose a neural network architecture and associated planning algorithm that (1) learns a representation of the world useful for generating prospective futures after the application of high-level actions, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) uses this same representation to evaluate these actions and perform tree search to find a sequence of high-level actions in a new environment.

Imitation Learning Robot Task Planning

On the Implicit Bias of Dropout

no code implementations ICML 2018 Poorya Mianjy, Raman Arora, Rene Vidal

Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings.

Streaming Kernel PCA with $\tilde{O}(\sqrt{n})$ Random Features

1 code implementation2 Aug 2018 Enayat Ullah, Poorya Mianjy, Teodor V. Marinov, Raman Arora

We study the statistical and computational aspects of kernel principal component analysis using random Fourier features and show that under mild assumptions, $O(\sqrt{n} \log n)$ features suffices to achieve $O(1/\epsilon^2)$ sample complexity.

Policy Regret in Repeated Games

no code implementations NeurIPS 2018 Raman Arora, Michael Dinitz, Teodor V. Marinov, Mehryar Mohri

We revisit the notion of policy regret and first show that there are online learning settings in which policy regret and external regret are incompatible: any sequence of play that achieves a favorable regret with respect to one definition must do poorly with respect to the other.

Learning from Multiview Correlations in Open-Domain Videos

no code implementations21 Nov 2018 Nils Holzenberger, Shruti Palaskar, Pranava Madhyastha, Florian Metze, Raman Arora

This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.

Representation Learning Retrieval

Differentially Private Robust Low-Rank Approximation

no code implementations NeurIPS 2018 Raman Arora, Vladimir Braverman, Jalaj Upadhyay

In this paper, we study the following robust low-rank matrix approximation problem: given a matrix $A \in \R^{n \times d}$, find a rank-$k$ matrix $B$, while satisfying differential privacy, such that $ \norm{ A - B }_p \leq \alpha \mathsf{OPT}_k(A) + \tau,$ where $\norm{ M }_p$ is the entry-wise $\ell_p$-norm and $\mathsf{OPT}_k(A):=\min_{\mathsf{rank}(X) \leq k} \norm{ A - X}_p$.

Streaming Kernel PCA with \tilde{O}(\sqrt{n}) Random Features

no code implementations NeurIPS 2018 Md Enayat Ullah, Poorya Mianjy, Teodor Vanislavov Marinov, Raman Arora

We study the statistical and computational aspects of kernel principal component analysis using random Fourier features and show that under mild assumptions, $O(\sqrt{n} \log n)$ features suffices to achieve $O(1/\epsilon^2)$ sample complexity.

Communication-efficient distributed SGD with Sketching

2 code implementations NeurIPS 2019 Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Vladimir Braverman, Ion Stoica, Raman Arora

Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time.

On Dropout and Nuclear Norm Regularization

1 code implementation28 May 2019 Poorya Mianjy, Raman Arora

We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss.

Bandits with Feedback Graphs and Switching Costs

no code implementations NeurIPS 2019 Raman Arora, Teodor V. Marinov, Mehryar Mohri

We give a new algorithm whose regret guarantee depends only on the domination number of the graph.

counterfactual

Efficient Convex Relaxations for Streaming PCA

1 code implementation NeurIPS 2019 Raman Arora, Teodor Vanislavov Marinov

We revisit two algorithms, matrix stochastic gradient (MSG) and $\ell_2$-regularized MSG (RMSG), that are instances of stochastic gradient descent (SGD) on a convex relaxation to principal component analysis (PCA).

Multiview Representation Learning for a Union of Subspaces

no code implementations30 Dec 2019 Nils Holzenberger, Raman Arora

Canonical correlation analysis (CCA) is a popular technique for learning representations that are maximally correlated across multiple views in data.

Representation Learning

Dropout: Explicit Forms and Capacity Control

no code implementations ICLR 2020 Raman Arora, Peter Bartlett, Poorya Mianjy, Nathan Srebro

In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks.

BIG-bench Machine Learning Matrix Completion

Corralling Stochastic Bandit Algorithms

no code implementations16 Jun 2020 Raman Arora, Teodor V. Marinov, Mehryar Mohri

We study the problem of corralling stochastic bandit algorithms, that is combining multiple bandit algorithms designed for a stochastic environment, with the goal of devising a corralling algorithm that performs almost as well as the best base algorithm.

Is Network the Bottleneck of Distributed Training?

1 code implementation17 Jun 2020 Zhen Zhang, Chaokun Chang, Haibin Lin, Yida Wang, Raman Arora, Xin Jin

As such, we advocate that the real challenge of distributed training is for the network community to develop high-performance network transport to fully utilize the network capacity and achieve linear scale-out.

FetchSGD: Communication-Efficient Federated Learning with Sketching

no code implementations15 Jul 2020 Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, Raman Arora

A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch.

Federated Learning

On Convergence and Generalization of Dropout Training

no code implementations NeurIPS 2020 Poorya Mianjy, Raman Arora

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations.

Machine Unlearning via Algorithmic Stability

no code implementations25 Feb 2021 Enayat Ullah, Tung Mai, Anup Rao, Ryan Rossi, Raman Arora

Our key contribution is the design of corresponding efficient unlearning algorithms, which are based on constructing a (maximal) coupling of Markov chains for the noisy SGD procedure.

Machine Unlearning

A Risk-Sensitive Policy Gradient Method

no code implementations29 Sep 2021 Jared Markowitz, Ryan Gardner, Ashley Llorens, Raman Arora, I-Jeng Wang

Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating a policy.

Decision Making

Differentially Private Generalized Linear Models Revisited

no code implementations6 May 2022 Raman Arora, Raef Bassily, Cristóbal Guzmán, Michael Menart, Enayat Ullah

For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) $\Theta\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^*\Vert}{\sqrt{n\epsilon}},\frac{\sqrt{\text{rank}}\Vert w^*\Vert}{n\epsilon}\right\}\right)$, where $\text{rank}$ is the rank of the design matrix.

Model Selection

Faster Rates of Convergence to Stationary Points in Differentially Private Optimization

no code implementations2 Jun 2022 Raman Arora, Raef Bassily, Tomás González, Cristóbal Guzmán, Michael Menart, Enayat Ullah

We provide a new efficient algorithm that finds an $\tilde{O}\big(\big[\frac{\sqrt{d}}{n\varepsilon}\big]^{2/3}\big)$-stationary point in the finite-sum setting, where $n$ is the number of samples.

Stochastic Optimization

Adversarial Robustness is at Odds with Lazy Training

no code implementations18 Jun 2022 Yunjuan Wang, Enayat Ullah, Poorya Mianjy, Raman Arora

Recent works show that adversarial examples exist for random neural networks [Daniely and Schacham, 2020] and that these examples can be found using a single step of gradient ascent [Bubeck et al., 2021].

Adversarial Robustness Learning Theory

A Risk-Sensitive Approach to Policy Optimization

1 code implementation19 Aug 2022 Jared Markowitz, Ryan W. Gardner, Ashley Llorens, Raman Arora, I-Jeng Wang

Without cost constraints, we find that pessimistic risk profiles can be used to reduce cost while improving total reward accumulation.

Decision Making

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

no code implementations23 Nov 2022 Thanh Nguyen-Tang, Ming Yin, Sunil Gupta, Svetha Venkatesh, Raman Arora

To the best of our knowledge, these are the first $\tilde{\mathcal{O}}(\frac{1}{K})$ bound and absolute zero sub-optimality bound respectively for offline RL with linear function approximation from adaptive data with partial coverage.

Offline RL reinforcement-learning +1

VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximation

1 code implementation24 Feb 2023 Thanh Nguyen-Tang, Raman Arora

We corroborate the statistical and computational efficiency of VIPeR with an empirical evaluation on a wide set of synthetic and real-world datasets.

Computational Efficiency Offline RL +3

From Adaptive Query Release to Machine Unlearning

no code implementations20 Jul 2023 Enayat Ullah, Raman Arora

We give efficient unlearning algorithms for linear and prefix-sum query classes.

Machine Unlearning

Differentially Private Non-Convex Optimization under the KL Condition with Optimal Rates

no code implementations22 Nov 2023 Michael Menart, Enayat Ullah, Raman Arora, Raef Bassily, Cristóbal Guzmán

We further show that, without assuming the KL condition, the same gradient descent algorithm can achieve fast convergence to a stationary point when the gradient stays sufficiently large during the run of the algorithm.

On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond

no code implementations6 Jan 2024 Thanh Nguyen-Tang, Raman Arora

This result is surprising, given that the prior work suggested an unfavorable sample complexity of the RO-based algorithm compared to the VS-based algorithm, whereas posterior sampling is rarely considered in offline RL due to its explorative nature.

Decision Making Offline RL +1

DART: A Principled Approach to Adversarially Robust Unsupervised Domain Adaptation

no code implementations16 Feb 2024 Yunjuan Wang, Hussein Hazimeh, Natalia Ponomareva, Alexey Kurakin, Ibrahim Hammoud, Raman Arora

To address this challenge, we first establish a generalization bound for the adversarial target loss, which consists of (i) terms related to the loss on the data, and (ii) a measure of worst-case domain divergence.

Adversarial Robustness Unsupervised Domain Adaptation

Offline Multitask Representation Learning for Reinforcement Learning

no code implementations18 Mar 2024 Haque Ishfaq, Thanh Nguyen-Tang, Songtao Feng, Raman Arora, Mengdi Wang, Ming Yin, Doina Precup

We study offline multitask representation learning in reinforcement learning (RL), where a learner is provided with an offline dataset from different tasks that share a common representation and is asked to learn the shared representation.

reinforcement-learning Reinforcement Learning (RL) +1

Communication-Efficient Federated Learning with Sketching

no code implementations ICML 2020 Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Vladimir Braverman, Joseph Gonzalez, Ion Stoica, Raman Arora

A key insight in the design of FedSketchedSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch.

Federated Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.