Search Results for author:
Found 40 papers, 3 papers with code
Beating Adversarial Low-Rank MDPs with Unknown Transition and Bandit Feedback
First, we improve the $poly(d, A, H)T^{5/6}$ regret bound of Zhao et al. (2024) to $poly(d, A, H)T^{2/3}$ for the full-information unknown transition setting, where d is the rank of the transitions, A is the number of actions, H is the horizon length, and T is the number of episodes.
How Does Variance Shape the Regret in Contextual Bandits?
We show that a regret of $\Omega(\sqrt{d_\text{elu}\Lambda}+d_\text{elu})$ is unavoidable when $\sqrt{d_\text{elu}\Lambda}+d_\text{elu}\leq\sqrt{AT}$.
Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification
In this work, we compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the action chosen by the learner, and weak corruption, where the corruption level does not depend on the action chosen by the learner.
Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data
We revisit the problem of offline reinforcement learning with value function realizability but without Bellman completeness.
On Tractable $Φ$-Equilibria in Non-Concave Games
While Online Gradient Descent and other no-regret learning procedures are known to efficiently converge to a coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this is not the case when utilities are non-concave -- a common scenario in machine learning applications involving strategies parameterized by deep neural networks, or when agents' utilities are computed by neural networks, or both.
Near-Optimal Policy Optimization for Correlated Equilibrium in General-Sum Markov Games
In this paper, we improve both results significantly by providing an uncoupled policy optimization algorithm that attains a near-optimal $\tilde{O}(T^{-1})$ convergence rate for computing a correlated equilibrium.
Towards Optimal Regret in Adversarial Linear MDPs with Bandit Feedback
The first algorithm, although computationally inefficient, ensures a regret of $\widetilde{\mathcal{O}}\left(\sqrt{K}\right)$, where $K$ is the number of episodes.
Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs
To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy.
A Blackbox Approach to Best of Both Worlds in Bandits and Beyond
Best-of-both-worlds algorithms for online learning which achieve near-optimal regret in both the adversarial and the stochastic regimes have received growing attention recently.
Best of Both Worlds Policy Optimization
Then we show that under known transitions, we can further obtain a first-order regret bound in the adversarial regime by leveraging the log-barrier regularizer.
Refined Regret for Adversarial MDPs with Linear Function Approximation
This analysis allows the loss estimators to be arbitrarily negative and might be of independent interest.
A Unified Algorithm for Stochastic Path Problems
Our regret bound matches the best known results for the well-studied special case of stochastic shortest path (SSP) with all non-positive rewards.
Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
Large-scale machine learning systems often involve data distributed across a collection of users.
Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence
When there is no uncertainty in the gradient evaluation, we show that our algorithm finds an $\epsilon$-Nash equilibrium with $O(1/\epsilon^2)$ iteration complexity which does not explicitly depend on the state space size.
Multi-agent Reinforcement Learning
Policy Gradient Methods
+1
Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure
For the bandit setting, we propose a hierarchical bandit algorithm that achieves a near-optimal gap-independent regret of $\widetilde{\mathcal{O}}(\sqrt{ABT})$ and a near-optimal gap-dependent regret of $\mathcal{O}(\log(T))$, where $A$ and $B$ are the numbers of actions of the leader and the follower, respectively, and $T$ is the number of steps.
A Model Selection Approach for Corruption Robust Reinforcement Learning
We develop a model selection approach to tackle reinforcement learning with adversarial corruption in both transition and reward.
Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonuses
When a simulator is unavailable, we further consider a linear MDP setting and obtain $\widetilde{\mathcal{O}}({T}^{14/15})$ regret, which is the first result for linear MDPs with adversarial losses and bandit feedback.
Achieving Near Instance-Optimality and Minimax-Optimality in Stochastic and Adversarial Linear Bandits Simultaneously
In this work, we develop linear bandit algorithms that automatically adapt to different environments.
Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach
Specifically, in most cases our algorithm achieves the optimal dynamic regret $\widetilde{\mathcal{O}}(\min\{\sqrt{LT}, \Delta^{1/3}T^{2/3}\})$ where $T$ is the number of rounds and $L$ and $\Delta$ are the number and amount of changes of the world respectively, while previous works only obtain suboptimal bounds and/or require the knowledge of $L$ and $\Delta$.
Last-iterate Convergence of Decentralized Optimistic Gradient Descent/Ascent in Infinite-horizon Competitive Markov Games
We study infinite-horizon discounted two-player zero-sum Markov games, and develop a decentralized algorithm that provably converges to the set of Nash equilibria under self-play.
Impossible Tuning Made Possible: A New Expert Algorithm and Its Applications
We resolve the long-standing "impossible tuning" issue for the classic expert problem and show that, it is in fact possible to achieve regret $O\left(\sqrt{(\ln d)\sum_t \ell_{t, i}^2}\right)$ simultaneously for all expert $i$ in a $T$-round $d$-expert problem where $\ell_{t, i}$ is the loss for expert $i$ in round $t$.
Minimax Regret for Stochastic Shortest Path with Adversarial Costs and Known Transition
We study the stochastic shortest path problem with adversarial costs and known transition, and show that the minimax regret is $\widetilde{O}(\sqrt{DT^\star K})$ and $\widetilde{O}(\sqrt{DT^\star SA K})$ for the full-information setting and the bandit feedback setting respectively, where $D$ is the diameter, $T^\star$ is the expected hitting time of the optimal policy, $S$ is the number of states, $A$ is the number of actions, and $K$ is the number of episodes.
Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation
We develop several new algorithms for learning Markov Decision Processes in an infinite-horizon average-reward setting with linear function approximation.
Linear Last-iterate Convergence in Constrained Saddle-point Optimization
Specifically, for OMWU in bilinear games over the simplex, we show that when the equilibrium is unique, linear last-iterate convergence is achieved with a learning rate whose value is set to a universal constant, improving the result of (Daskalakis & Panageas, 2019b) under the same assumption.
Bias no more: high-probability data-dependent regret bounds for adversarial bandits and MDPs
We develop a new approach to obtaining high probability regret bounds for online learning with bandit feedback against an adaptive adversary.
A Model-free Learning Algorithm for Infinite-horizon Average-reward MDPs with Near-optimal Regret
Recently, model-free reinforcement learning has attracted research attention due to its simplicity, memory and computation efficiency, and the flexibility to combine with function approximation.
Federated Residual Learning
We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model.
Adversarial Online Learning with Changing Action Sets: Efficient Algorithms with Approximate Regret Bounds
We revisit the problem of online learning with sleeping experts/bandits: in each time step, only a subset of the actions are available for the algorithm to choose from (and learn about).
Taking a hint: How to leverage loss predictors in contextual bandits?
We initiate the study of learning in contextual bandits with the help of loss predictors.
Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes
Model-free reinforcement learning is known to be memory and computation efficient and more amendable to large scale problems.
Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise.
Bandit Multiclass Linear Classification: Efficient Algorithms for the Separable Case
Under the more challenging weak linear separability condition, we design an efficient algorithm with a mistake bound of $\min (2^{\widetilde{O}(K \log^2 (1/\gamma))}, 2^{\widetilde{O}(\sqrt{1/\gamma} \log K)})$.
A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free
We propose the first contextual bandit algorithm that is parameter-free, efficient, and optimal in terms of dynamic regret.
Improved Path-length Regret Bounds for Bandits
We study adaptive regret bounds in terms of the variation of the losses (the so-called path-length bounds) for both multi-armed bandit and more generally linear bandit.
Beating Stochastic and Adversarial Semi-bandits Optimally and Simultaneously
We develop the first general semi-bandit algorithm that simultaneously achieves $\mathcal{O}(\log T)$ regret for stochastic environments and $\mathcal{O}(\sqrt{T})$ regret for adversarial environments without knowledge of the regime or the number of rounds $T$.
Efficient Online Portfolio with Logarithmic Regret
We study the decades-old problem of online portfolio management and propose the first algorithm with logarithmic regret that is not based on Cover's Universal Portfolio algorithm and admits much faster implementation.
More Adaptive Algorithms for Adversarial Bandits
We develop a novel and generic algorithm for the adversarial multi-armed bandit problem (or more generally the combinatorial semi-bandit problem).
Online Reinforcement Learning in Stochastic Games
We study online reinforcement learning in average-reward stochastic games (SGs).
Tracking the Best Expert in Non-stationary Stochastic Environments
We study the dynamic regret of multi-armed bandit and experts problem in non-stationary stochastic environments.
Efficient Contextual Bandits in Non-stationary Worlds
In this work, we develop several efficient contextual bandit algorithms for non-stationary environments by equipping existing methods for i. i. d.
