Search Results for author:
Found 19 papers, 1 papers with code
Convergence for Natural Policy Gradient on Infinite-State Average-Reward Markov Decision Processes
In the reinforcement learning (RL) context, a variety of algorithms have been developed to learn and optimize these MDPs.
Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise
SA appears in many areas such as optimization and Reinforcement Learning (RL).
Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise
To demonstrate the applicability of our theoretical results, we use them to provide maximal concentration bounds for a large class of reinforcement learning algorithms, including but not limited to on-policy TD-learning with linear function approximation, off-policy TD-learning with generalized importance sampling factors, and $Q$-learning.
An Approximate Policy Iteration Viewpoint of Actor-Critic Algorithms
Combining the geometric convergence of the actor with the finite-sample analysis of the critic, we establish for the first time an overall $\mathcal{O}(\epsilon^{-2})$ sample complexity for finding an optimal policy (up to a function approximation error) using policy-based methods under off-policy sampling and linear function approximation.
Federated Reinforcement Learning: Linear Speedup Under Markovian Sampling
To obtain these results, we show that federated TD and Q-learning are special cases of a general framework for federated stochastic approximation with Markovian noise, and we leverage this framework to provide a unified convergence analysis that applies to all the algorithms.
Target Network and Truncation Overcome The Deadly Triad in $Q$-Learning
$Q$-learning with function approximation is one of the most empirically successful while theoretically mysterious reinforcement learning (RL) algorithms, and was identified in Sutton (1999) as one of the most important theoretical open problems in the RL community.
Finite Sample Analysis of Average-Reward TD Learning and $Q$-Learning
The focus of this paper is on sample complexity guarantees of average-reward reinforcement learning algorithms, which are known to be more challenging to study than their discounted-reward counterparts.
Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterization
In this work, we study the asymptotic behavior of the appropriately scaled stationary distribution, in the limit when the constant stepsize goes to zero.
Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators
Our key step is to show that the generalized Bellman operator is simultaneously a contraction mapping with respect to a weighted $\ell_p$-norm for each $p$ in $[1,\infty)$, with a common contraction factor.
Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximation
In this paper, we develop a novel variant of off-policy natural actor-critic algorithm with linear function approximation and we establish a sample complexity of $\mathcal{O}(\epsilon^{-3})$, outperforming all the previously known convergence bounds of such algorithms.
On the Linear convergence of Natural Policy Gradient Algorithm
We further improve this convergence result by introducing a variant of Natural Policy Gradient with adaptive step sizes.
Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
In this paper, we provide finite-sample convergence guarantees for an off-policy variant of the natural actor-critic (NAC) algorithm based on Importance Sampling.
A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variants
As a by-product, by analyzing the convergence bounds of $n$-step TD and TD$(\lambda)$, we provide theoretical insights into the bias-variance trade-off, i. e., efficiency of bootstrapping in RL.
Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithm
In this paper, we characterize the \emph{global} convergence of an online natural actor-critic algorithm in the tabular setting using a single trajectory of samples.
Near Optimal Control in Ride Hailing Platforms with Strategic Servers
We then study the system under a large market regime in which the arrival rates are scaled by $\eta$ and present a probabilistic two-price policy and a max-weight matching policy which results in a net profit-loss of at most $O(\eta^{1/3})$.
Optimization and Control Probability
Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes
In particular, we use it to establish the first-known convergence rate of the V-trace algorithm for off-policy TD-learning.
Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximation
Our main contribution is to provide a finite-analysis on the performance of this distributed {\sf TD}$(\lambda)$ algorithm for both constant and time-varying step sizes.
Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
To demonstrate the generality of our theoretical results on Markovian SA, we use it to derive the finite-sample bounds of the popular $Q$-learning with linear function approximation algorithm, under a condition on the behavior policy.
Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
In this problem, a group of agents works cooperatively to evaluate the value function for the global discounted accumulative reward problem, which is composed of local rewards observed by the agents.
Optimization and Control
