Search Results for author:
Found 72 papers, 9 papers with code
Learning Adversarial Markov Decision Processes with Bandit Feedback and Unknown Transition
We consider the task of learning in episodic finite-horizon Markov decision processes with an unknown transition function, bandit feedback, and adversarial losses.
Tuning-Free Stochastic Optimization
For the task of finding a stationary point of a smooth and potentially nonconvex function, we give a variant of SGD that matches the best-known high-probability convergence rate for tuned SGD at only an additional polylogarithmic cost.
Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift
This paper addresses this fundamental question by proving that, surprisingly, classical Maximum Likelihood Estimation (MLE) purely using source data (without any modification) achieves the minimax optimality for covariate shift under the well-specified setting.
ZeroSwap: Data-driven Optimal Market Making in DeFi
This ensures that the market maker balances losses to informed traders with profits from noise traders.
Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online.
Is RLHF More Difficult than Standard RL?
This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs.
PEBO-SLAM: Observer design for visual inertial SLAM with convergence guarantees
This paper introduces a new linear parameterization to the problem of visual inertial simultaneous localization and mapping (VI-SLAM) -- without any approximation -- for the case only using information from a single monocular camera and an inertial measurement unit.
DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method
This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients).
Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL
While policy optimization algorithms have played an important role in recent empirical success of Reinforcement Learning (RL), the existing theoretical understanding of policy optimization remains rather limited -- they are either restricted to tabular MDPs or suffer from highly suboptimal sample complexity, especial in online RL where exploration is necessary.
Learning a Universal Human Prior for Dexterous Manipulation from Human Preference
Generating human-like behavior on robots is a great challenge especially in dexterous manipulation tasks with robotic hands.
On the Provable Advantage of Unsupervised Pretraining
Unsupervised pretraining, which learns a useful representation using a large amount of unlabeled data to facilitate the learning of downstream tasks, is a critical component of modern large-scale machine learning systems.
Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximation
A unique challenge in Multi-Agent Reinforcement Learning (MARL) is the curse of multiagency, where the description length of the game as well as the complexity of many existing learning algorithms scale exponentially with the number of agents.
Efficient displacement convex optimization with particle gradient descent
Particle gradient descent, which uses particles to represent a probability measure and performs gradient descent on particles in parallel, is widely used to optimize functions of probability measures.
Representation Learning for General-sum Low-rank Markov Games
To our best knowledge, this is the first sample-efficient algorithm for multi-agent general-sum Markov games that incorporates (non-linear) function approximation.
Provable Sim-to-real Transfer in Continuous Domain with Partial Observations
Sim-to-real transfer trains RL agents in the simulated environments and then deploys them in the real world.
Learning Rationalizable Equilibria in Multiplayer Games
This paper develops the first line of efficient algorithms for learning rationalizable Coarse Correlated Equilibria (CCE) and Correlated Equilibria (CE) whose sample complexities are polynomial in all problem parameters including the number of players.
Optimistic MLE -- A Generic Model-based Algorithm for Partially Observable Sequential Decision Making
We prove that OMLE learns the near-optimal policies of an enormously rich class of sequential decision making problems in a polynomial number of samples.
Faster federated optimization under second-order similarity
Federated learning (FL) is a subfield of machine learning where multiple clients try to collaboratively learn a model over a network under communication constraints.
A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games
This paper proposes new, end-to-end deep reinforcement learning algorithms for learning two-player zero-sum Markov games.
Sample-Efficient Reinforcement Learning of Partially Observable Markov Games
This paper considers the challenging tasks of Multi-Agent Reinforcement Learning (MARL) under partial observability, where each agent only sees her own individual observations and actions that reveal incomplete information about the underlying state of system.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descent
A conceptually appealing approach for learning Extensive-Form Games (EFGs) is to convert them to Normal-Form Games (NFGs).
When Is Partially Observable Reinforcement Learning Not Scary?
Applications of Reinforcement Learning (RL), in which agents learn to make a sequence of decisions despite lacking complete information about the latent states of the controlled system, that is, they act under partial observability of the states, are ubiquitous.
Partially Observable Reinforcement Learning
reinforcement-learning
+1
Learning Markov Games with Adversarial Opponents: Efficient Algorithms and Fundamental Limits
When the policies of the opponents are not revealed, we prove a statistical hardness result even in the most favorable scenario when both above conditions are true.
Provable Reinforcement Learning with a Short-Term Memory
Real-world sequential decision making problems commonly involve partial observability, which requires the agent to maintain a memory of history in order to infer the latent states, plan and make good decisions.
Near-Optimal Learning of Extensive-Form Games with Imperfect Information
This improves upon the best known sample complexity of $\widetilde{\mathcal{O}}((X^2A+Y^2B)/\varepsilon^2)$ by a factor of $\widetilde{\mathcal{O}}(\max\{X, Y\})$, and matches the information-theoretic lower bound up to logarithmic factors.
Globally convergent visual-feature range estimation with biased inertial measurements
The design of a globally convergent position observer for feature points from visual information is a challenging problem, especially for the case with only inertial measurements and without assumptions of uniform observability, which remained open for a long time.
V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL
We design a new class of fully decentralized algorithms -- V-learning, which provably learns Nash equilibria (in the two-player zero-sum setting), correlated equilibria and coarse correlated equilibria (in the multiplayer general-sum setting) in a number of samples that only scales with $\max_{i\in[m]} A_i$, where $A_i$ is the number of actions for the $i^{\rm th}$ player.
Understanding Domain Randomization for Sim-to-real Transfer
Reinforcement learning encounters many challenges when applied directly in the real world.
A Simple Reward-free Approach to Constrained Reinforcement Learning
In constrained reinforcement learning (RL), a learning agent seeks to not only optimize the overall reward but also satisfy the additional safety, diversity, or budget constraints.
The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
Modern reinforcement learning (RL) commonly engages practical problems with large state spaces, where function approximation must be deployed to approximate either the value function or the policy.
Minimax Optimization with Smooth Algorithmic Adversaries
This paper considers minimax optimization $\min_x \max_y f(x, y)$ in the challenging setting where $f$ can be both nonconvex in $x$ and nonconcave in $y$.
Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning
Concretely, we view the Bellman error as a surrogate loss for the optimality gap, and prove the followings: (1) In double sampling regime, the excess risk of Empirical Risk Minimizer (ERM) is bounded by the Rademacher complexity of the function class.
Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Games
Real world applications such as economics and policy making often involve solving multi-agent games with two unique features: (1) The agents are inherently asymmetric and partitioned into leaders and followers; (2) The agents have different reward functions, thus the game is general-sum.
Near-optimal Representation Learning for Linear Bandits and Linear RL
This paper studies representation learning for multi-task linear bandits and multi-task episodic RL with linear value function approximation.
A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network
We show that as long as the loss is already lower than a threshold (polynomial in relevant parameters), all student neurons in an over-parameterized two-layer neural network will converge to one of teacher neurons, and the loss will go to 0.
Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
Finding the minimal structural assumptions that empower sample-efficient learning is one of the most important research directions in Reinforcement Learning (RL).
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States
We propose a novel setting for reinforcement learning that combines two common real-world difficulties: presence of observations (such as camera images) and factored states (such as location of objects).
Provably Efficient Reinforcement Learning with Kernel and Neural Function Approximations
Reinforcement learning (RL) algorithms combined with modern function approximators such as kernel functions and deep neural networks have achieved significant empirical successes in large-scale application problems with a massive number of states.
On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces
The classical theory of reinforcement learning (RL) has focused on tabular and linear representations of value functions.
A Sharp Analysis of Model-based Reinforcement Learning with Self-Play
However, for multi-agent reinforcement learning in Markov games, the current best known sample complexity for model-based algorithms is rather suboptimal and compares unfavorably against recent model-free approaches.
Model-based Reinforcement Learning
Multi-agent Reinforcement Learning
+2
Sample-Efficient Reinforcement Learning of Undercomplete POMDPs
Partial observability is a common challenge in many reinforcement learning applications, which requires an agent to maintain memory, infer latent states, and integrate this past information into exploration.
Near-Optimal Reinforcement Learning with Self-Play
This paper considers the problem of designing optimal algorithms for reinforcement learning in two-player zero-sum games.
On the Theory of Transfer Learning: The Importance of Task Diversity
Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j \circ h$ in a general function class $\mathcal{F} \circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$.
Provable Meta-Learning of Linear Representations
In this paper, we focus on the problem of multi-task linear regression -- in which multiple linear regression models share a common, low-dimensional linear representation.
Provable Self-Play Algorithms for Competitive Reinforcement Learning
We introduce a self-play algorithm---Value Iteration with Upper/Lower Confidence Bound (VI-ULCB)---and show that it achieves regret $\tilde{\mathcal{O}}(\sqrt{T})$ after playing $T$ steps of the game, where the regret is measured by the agent's performance against a \emph{fully adversarial} opponent who can exploit the agent's strategy at \emph{any} step.
Reward-Free Exploration for Reinforcement Learning
We give an efficient algorithm that conducts $\tilde{\mathcal{O}}(S^2A\mathrm{poly}(H)/\epsilon^2)$ episodes of exploration and returns $\epsilon$-suboptimal policies for an arbitrary number of reward functions.
Near-Optimal Algorithms for Minimax Optimization
This paper presents the first algorithm with $\tilde{O}(\sqrt{\kappa_{\mathbf x}\kappa_{\mathbf y}})$ gradient complexity, matching the lower bound up to logarithmic factors.
Provably Efficient Exploration in Policy Optimization
While policy-based reinforcement learning (RL) achieves tremendous successes in practice, it is significantly less understood in theory, especially compared with value-based RL.
Learning Adversarial MDPs with Bandit Feedback and Unknown Transition
We consider the problem of learning in episodic finite-horizon Markov decision processes with an unknown transition function, bandit feedback, and adversarial losses.
Provably Efficient Reinforcement Learning with Linear Function Approximation
Modern Reinforcement Learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy.
On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
We consider nonconvex-concave minimax problems, $\min_{\mathbf{x}} \max_{\mathbf{y} \in \mathcal{Y}} f(\mathbf{x}, \mathbf{y})$, where $f$ is nonconvex in $\mathbf{x}$ but concave in $\mathbf{y}$ and $\mathcal{Y}$ is a convex and bounded set.
On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial.
A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm
In this note, we derive concentration inequalities for random vectors with subGaussian norm (a generalization of both subGaussian random vectors and norm bounded random vectors), which are tight up to logarithmic factors.
What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
Minimax optimization has found extensive applications in modern machine learning, in settings such as generative adversarial networks (GANs), adversarial training and multi-agent reinforcement learning.
BIG-bench Machine Learning
Multi-agent Reinforcement Learning
Sampling Can Be Faster Than Optimization
Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years.
Is Q-learning Provably Efficient?
We prove that, in an episodic MDP setting, Q-learning with UCB exploration achieves regret $\tilde{O}(\sqrt{H^3 SAT})$, where $S$ and $A$ are the numbers of states and actions, $H$ is the number of steps per episode, and $T$ is the total number of steps.
Stability and Convergence Trade-off of Iterative Optimization Algorithms
Applying existing stability upper bounds for the gradient methods in our trade-off framework, we obtain lower bounds matching the well-established convergence upper bounds up to constants for these algorithms and conjecture similar lower bounds for NAG and HB.
On the Local Minima of the Empirical Risk
Our objective is to find the $\epsilon$-approximate local minima of the underlying function $F$ while avoiding the shallow local minima---arising because of the tolerance $\nu$---which exist only in $f$.
Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting.
Stochastic Cubic Regularization for Fast Nonconvex Optimization
This paper proposes a stochastic variant of a classic algorithm---the cubic-regularized Newton method [Nesterov and Polyak 2006].
Gradient Descent Can Take Exponential Time to Escape Saddle Points
Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape.
No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
In this paper we develop a new framework that captures the common landscape underlying the common non-convex low-rank matrix problems including matrix sensing, matrix completion and robust PCA.
How to Escape Saddle Points Efficiently
This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i. e., it is almost "dimension-free").
Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences
Our first main result shows that the population likelihood function has bad local maxima even in the special case of equally-weighted mixtures of well-separated and spherical Gaussians.
Faster Eigenvector Computation via Shift-and-Invert Preconditioning
We give faster algorithms and improved sample complexities for estimating the top eigenvector of a matrix $\Sigma$ -- i. e. computing a unit vector $x$ such that $x^T \Sigma x \ge (1-\epsilon)\lambda_1(\Sigma)$: Offline Eigenvector Estimation: Given an explicit $A \in \mathbb{R}^{n \times d}$ with $\Sigma = A^TA$, we show how to compute an $\epsilon$ approximate top eigenvector in time $\tilde O([nnz(A) + \frac{d*sr(A)}{gap^2} ]* \log 1/\epsilon )$ and $\tilde O([\frac{nnz(A)^{3/4} (d*sr(A))^{1/4}}{\sqrt{gap}} ] * \log 1/\epsilon )$.
Provable Efficient Online Matrix Completion via Non-convex Stochastic Gradient Descent
While existing algorithms are efficient for the offline setting, they could be highly inefficient for the online setting.
Efficient Algorithms for Large-scale Generalized Eigenvector Computation and Canonical Correlation Analysis
Our algorithm is linear in the input size and the number of components $k$ up to a $\log(k)$ factor.
Streaming PCA: Matching Matrix Bernstein and Near-Optimal Finite Sample Guarantees for Oja's Algorithm
This work provides improved guarantees for streaming principle component analysis (PCA).
Robust Shift-and-Invert Preconditioning: Faster and More Sample Efficient Algorithms for Eigenvector Computation
Combining our algorithm with previous work to initialize $x_0$, we obtain a number of improved sample complexity and runtime results.
Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points.
Differentially Private Data Releasing for Smooth Queries with Synthetic Database Output
We develop an $\epsilon$-differentially private mechanism for the class of $K$-smooth queries.
Dimensionality Dependent PAC-Bayes Margin Bound
We show that our bound is strictly sharper than a previously well-known PAC-Bayes margin bound if the feature space is of finite dimension; and the two bounds tend to be equivalent as the dimension goes to infinity.
