Search Results for author:
Found 59 papers, 8 papers with code
Provable Partially Observable Reinforcement Learning with Privileged Information
Partial observability of the underlying states generally presents significant challenges for reinforcement learning (RL).
Partially Observable Reinforcement Learning
reinforcement-learning
+2
Last-Iterate Convergence of Payoff-Based Independent Learning in Zero-Sum Stochastic Games
In this paper, we consider two-player zero-sum matrix and stochastic games and develop learning dynamics that are payoff-based, convergent, rational, and symmetric between the two players.
RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions.
Do LLM Agents Have Regret? A Case Study in Online Learning and Games
To better understand the limits of LLM agents in these interactive environments, we propose to study their interactions in benchmark decision-making settings in online learning and game theory, through the performance metric of \emph{regret}.
Two-Timescale Q-Learning with Function Approximation in Zero-Sum Stochastic Games
Specifically, through a change of variable, we show that the update equation of the slow-timescale iterates resembles the classical smoothed best-response dynamics, where the regularized Nash gap serves as a valid Lyapunov function.
Robot Fleet Learning via Policy Merging
We show that FLEET-MERGE consolidates the behavior of policies trained on 50 tasks in the Meta-World environment, with good performance on nearly all training tasks at test time.
Partially Observable Multi-Agent Reinforcement Learning with Information Sharing
We establish concrete computational and sample complexities under several common structural assumptions of the model.}
Computational Efficiency
Multi-agent Reinforcement Learning
+3
Multi-Player Zero-Sum Markov Games with Networked Separable Interactions
We study a new class of Markov games, \emph(multi-player) zero-sum Markov Games} with \emph{Networked separable interactions} (zero-sum NMGs), to model the local interaction structure in non-cooperative multi-agent sequential decision-making.
Tackling Combinatorial Distribution Shift: A Matrix Completion Perspective
Focusing on the special case where the labels are given by bilinear embeddings into a Hilbert space $H$: $\mathbb{E}[z \mid x, y ]=\langle f_{\star}(x), g_{\star}(y)\rangle_{{H}}$, we aim to extrapolate to a test distribution domain that is $not$ covered in training, i. e., achieving bilinear combinatorial extrapolation.
Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs
To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy.
Learning to Extrapolate: A Transductive Approach
Machine learning systems, especially with overparameterized deep neural networks, can generalize to novel test instances drawn from the same distribution as the training data.
Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation
In contrast, existing works for Markov games with function approximation have sample complexity bounds scale with the size of the \emph{joint action space} when specialized to the canonical tabular Markov game setting, which is exponentially large in the number of agents.
Can Direct Latent Model Learning Solve Linear Quadratic Gaussian Control?
We study the task of learning state representations from potentially high-dimensional observations, with the goal of controlling an unknown partially observable system.
Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
In this work, we revisit the linear programming (LP) reformulation of Markov decision processes for offline RL, with the goal of developing algorithms with optimal $O(1/\sqrt{n})$ sample complexity, where $n$ is the sample size, under partial data coverage and general function approximation, and with favorable computational tractability.
An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
In this paper, we revisit and improve the convergence of policy gradient (PG), natural PG (NPG) methods, and their variance-reduced variants, under general smooth policy parametrizations.
Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergence
We also generalize the results to certain function approximation settings.
Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?
Decentralized learning has been advocated and widely deployed to make efficient use of distributed datasets, with an extensive focus on supervised learning (SL) problems.
Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies
Gradient-based methods have been widely used for system design and optimization in diverse application domains.
The Power of Regularization in Solving Extensive-Form Games
Second, we show that regularized counterfactual regret minimization (\texttt{Reg-CFR}), with a variant of optimistic mirror descent algorithm as regret-minimizer, can achieve $O(1/T^{1/4})$ best-iterate, and $O(1/T^{3/4})$ average-iterate convergence rate for finding NE in EFGs.
What is a Good Metric to Study Generalization of Minimax Learners?
Minimax optimization has served as the backbone of many machine learning (ML) problems.
Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs
We study sequential decision making problems aimed at maximizing the expected total reward while satisfying a constraint on the expected total utility.
Byzantine-Robust Online and Offline Distributed Reinforcement Learning
We consider a distributed reinforcement learning setting where multiple agents separately explore the environment and communicate their experiences through a central server.
The Complexity of Markov Equilibrium in Stochastic Games
Previous work for learning Markov CCE policies all required exponential time and sample complexity in the number of players.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Globally Convergent Policy Search over Dynamic Filters for Output Estimation
In this paper, we provide a new perspective on this challenging problem based on the notion of $\textit{informativity}$, which intuitively requires that all components of a filter's internal state are representative of the true state of the underlying dynamical system.
Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence
When there is no uncertainty in the gradient evaluation, we show that our algorithm finds an $\epsilon$-Nash equilibrium with $O(1/\epsilon^2)$ iteration complexity which does not explicitly depend on the state space size.
Multi-agent Reinforcement Learning
Policy Gradient Methods
+1
Do Differentiable Simulators Give Better Policy Gradients?
Differentiable simulators promise faster computation time for reinforcement learning by replacing zeroth-order gradient estimates of a stochastic objective with an estimate based on first-order gradients.
Independent Learning in Stochastic Games
We focus on the development of simple and independent learning dynamics for stochastic games: each agent is myopic and chooses best-response type actions to other agents' strategy without any coordination with her opponent.
On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning
Multi-agent reinforcement learning (MARL) algorithms often suffer from an exponential sample complexity dependence on the number of agents, a phenomenon known as \emph{the curse of multiagents}.
Decentralized Cooperative Multi-Agent Reinforcement Learning with Exploration
Many real-world applications of multi-agent reinforcement learning (RL), such as multi-robot navigation and decentralized control of cyber-physical systems, involve the cooperation of agents as a team with aligned objectives.
Decentralized Q-Learning in Zero-sum Markov Games
The key challenge in this decentralized setting is the non-stationarity of the environment from an agent's perspective, since both her own payoffs and the system evolution depend on the actions of other agents, and each agent adapts her policies simultaneously and independently.
Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates
We propose a novel joint-learning framework that can be implemented in a decentralized fashion, with generalization guarantees for certain function classes.
Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
Direct policy search serves as one of the workhorses in modern reinforcement learning (RL), and its applications in continuous control tasks have recently attracted increasing attention.
Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup
Asynchronous and parallel implementation of standard reinforcement learning (RL) algorithms is a key enabler of the tremendous success of modern RL.
Robust Multi-Agent Reinforcement Learning with Model Uncertainty
In contrast, we model the problem as a robust Markov game, where the goal of all agents is to find policies such that no agent has the incentive to deviate, i. e., reach some equilibrium point, which is also robust to the possible uncertainty of the MARL model.
Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
To the best of our knowledge, our work is the first to establish non-asymptotic convergence guarantees of policy-based primal-dual methods for solving infinite-horizon discounted CMDPs.
On the Stability and Convergence of Robust Adversarial Reinforcement Learning: A Case Study on Linear Quadratic Systems
We find: i) the conventional RARL framework (Pinto et al., 2017) can learn a destabilizing policy if the initial policy does not enjoy the robust stability property against the adversary; and ii) with robustly stabilizing initializations, our proposed double-loop RARL algorithm provably converges to the global optimal cost while maintaining robust stability on-the-fly.
Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Control
We consider model-free reinforcement learning (RL) in non-stationary Markov decision processes.
Near-Optimal Regret Bounds for Model-Free RL in Non-Stationary Episodic MDPs
We consider model-free reinforcement learning (RL) in non-stationary Markov decision processes (MDPs).
Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic Mean-Field Games
We propose an actor-critic algorithm to iteratively compute the mean-field equilibrium (MFE) of the LQ-MFG.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
This is in contrast to the usual reward-aware setting, with a $\tilde\Omega(|S|(|A|+|B|)(1-\gamma)^{-3}\epsilon^{-2})$ lower bound, where this model-based approach is near-optimal with only a gap on the $|A|,|B|$ dependence.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysis
Monte-Carlo planning, as exemplified by Monte-Carlo Tree Search (MCTS), has demonstrated remarkable performance in applications with finite spaces.
Policy Optimization for $\mathcal{H}_{2}$ Linear Control with $\mathcal{H}_{\infty}$ Robustness Guarantee: Implicit Regularization and Global Convergence
In this paper, we study the convergence theory of PO for $\mathcal{H}_{2}$ linear control with $\mathcal{H}_{\infty}$ robustness guarantee.
Information State Embedding in Partially Observable Cooperative Multi-Agent Reinforcement Learning
To enable the development of tractable algorithms, we introduce the concept of an information state embedding that serves to compress agents' histories.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Fully Asynchronous Policy Evaluation in Distributed Reinforcement Learning over Networks
This paper proposes a \emph{fully asynchronous} scheme for the policy evaluation problem of distributed reinforcement learning (DisRL) over directed peer-to-peer networks.
Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances
Multi-agent reinforcement learning (MARL) has long been a significant and everlasting research topic in both machine learning and control.
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Orthogonal to the existing reviews on MARL, we highlight several new angles and taxonomies of MARL theory, including learning in extensive-form games, decentralized MARL with networked agents, MARL in the mean-field regime, (non-)convergence of policy-based methods for learning in games, etc.
Non-Cooperative Inverse Reinforcement Learning
Through interacting with the more informed player, the less informed player attempts to both infer, and act according to, the true objective function.
Policy Optimization for $\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness Guarantee: Implicit Regularization and Global Convergence
In this paper, we study the convergence theory of PO for $\mathcal{H}_2$ linear control with $\mathcal{H}_\infty$-norm robustness guarantee.
Online Planning for Decentralized Stochastic Control with Partial History Sharing
To demonstrate the applicability of the model, we propose a novel collaborative intrusion response model, where multiple agents (defenders) possessing asymmetric information aim to collaboratively defend a computer network.
A Convergence Result for Regularized Actor-Critic Methods
In this paper, we present a probability one convergence proof, under suitable conditions, of a certain class of actor-critic algorithms for finding approximate solutions to entropy-regularized MDPs using the machinery of stochastic approximation.
A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning
This paper considers a distributed reinforcement learning problem in which a network of multiple agents aim to cooperatively maximize the globally averaged return through communication with only local neighbors.
Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
Under a further strict saddle points assumption, this result establishes convergence to essentially locally-optimal policies of the underlying problem, and thus bridges the gap in existing literature on the convergence of PG methods.
Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
To the best of our knowledge, this work appears to be the first one to investigate the optimization landscape of LQ games, and provably show the convergence of policy optimization methods to the Nash equilibria.
A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy.
Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning
This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners.
Finite-Sample Analysis For Decentralized Batch Multi-Agent Reinforcement Learning With Networked Agents
This work appears to be the first finite-sample analysis for batch MARL, a step towards rigorous theoretical understanding of general MARL algorithms in the finite-sample regime.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Distributed Learning of Average Belief Over Networks Using Sequential Observations
This paper addresses the problem of distributed learning of average belief with sequential observations, in which a network of $n>1$ agents aim to reach a consensus on the average value of their beliefs, by exchanging information only with their neighbors.
Convergent Reinforcement Learning with Function Approximation: A Bilevel Optimization Perspective
We study reinforcement learning algorithms with nonlinear function approximation in the online setting.
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
