Search Results for author:
Found 64 papers, 12 papers with code
Critic-Actor for Average Reward MDPs with Function Approximation: A Finite-Time Analysis
In recent years, there has been a lot of research work activity focused on carrying out asymptotic and non-asymptotic convergence analyses for two-timescale actor critic algorithms where the actor updates are performed on a timescale that is slower than that of the critic.
Approximate Linear Programming and Decentralized Policy Improvement in Cooperative Multi-agent Markov Decision Processes
In this work, we consider a `cooperative' multi-agent Markov decision process (MDP) involving m greater than 1 agents, where all agents are aware of the system model.
Finite Time Analysis of Constrained Actor Critic and Constrained Natural Actor Critic Algorithms
Actor Critic methods have found immense applications on a wide range of Reinforcement Learning tasks especially when the state-action space is large.
Energy Management in a Cooperative Energy Harvesting Wireless Sensor Network
In this paper, we consider the problem of finding an optimal energy management policy for a network of sensor nodes capable of harvesting their own energy and sharing it with other nodes in the network.
The Reinforce Policy Gradient Algorithm Revisited
This has advantages in the case of systems with infinite state and action spaces as it relax some of the regularity requirements that would otherwise be needed for proving convergence of the Reinforce algorithm.
A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks
We show, in theory and through experiments, that our algorithm updates have low variance, and the training loss reduces in a smooth manner.
Off-Policy Average Reward Actor-Critic with Deterministic Policy Search
In this work, we present both on-policy and off-policy deterministic policy gradient theorems for the average reward performance criterion.
A Cubic-regularized Policy Newton Algorithm for Reinforcement Learning
We consider the problem of control in the setting of reinforcement learning (RL), where model information is not available.
n-Step Temporal Difference Learning with Optimal n
We consider the problem of finding the optimal value of n in the n-step temporal difference (TD) learning algorithm.
Generalized Simultaneous Perturbation-based Gradient Search with Reduced Estimator Bias
We first present in detail unbalanced generalized simultaneous perturbation stochastic approximation (GSPSA) estimators and later present the balanced versions (B-GSPSA) of these.
Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark - the Open AI Safety Gym.
Actor-Critic or Critic-Actor? A Tale of Two Time Scales
We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale.
A policy gradient approach for Finite Horizon Constrained Markov Decision Processes
In many situations, finite horizon control problems are of interest and for such problems, the optimal policies are time-varying in general.
A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization
In this paper, we present a stochastic gradient algorithm for minimizing a smooth objective function that is an expectation over noisy cost samples, and only the latter are observed for any given parameter.
Reinforcement Learning for Task Specifications with Action-Constraints
We assume that the set of action sequences that are deemed unsafe and/or safe are given in terms of a finite-state automaton; and propose a supervisor that disables a subset of actions at every state of the MDP so that the constraints on action sequence are satisfied.
$N$-Timescale Stochastic Approximation: Stability and Convergence
This paper presents the first sufficient conditions that guarantee the stability and almost sure convergence of $N$-timescale stochastic approximation (SA) iterates for any $N\geq1$.
Schedule Based Temporal Difference Algorithms
However, the weights assigned to different $n$-step returns in TD($\lambda$), controlled by the parameter $\lambda$, decrease exponentially with increasing $n$.
Gradient Temporal Difference with Momentum: Stability and Convergence
Here, we consider Gradient TD algorithms with an additional heavy ball momentum term and provide choice of step size and momentum parameter that ensures almost sure convergence of these algorithms asymptotically.
Neural Network Compatible Off-Policy Natural Actor-Critic Algorithm
Learning optimal behavior from existing data is one of the most important problems in Reinforcement Learning (RL).
Attention Actor-Critic algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning
In this work, we consider the problem of computing optimal actions for Reinforcement Learning (RL) agents in a co-operative setting, where the objective is to optimize a common goal.
Robust Quadrupedal Locomotion on Sloped Terrains: A Linear Policy Approach
In particular, the parameters of the end-foot trajectories are shaped via a linear feedback policy that takes the torso orientation and the terrain slope as inputs.
Hindsight Experience Replay with Kronecker Product Approximate Curvature
Natural gradients solves these challenges by converging the model parameters better.
A reinforcement learning approach to hybrid control design
Second, we model a set of benchmark examples of hybrid control design problem in the proposed MDP framework.
Learning Stable Manoeuvres in Quadruped Robots from Expert Demonstrations
First, multiple simpler policies are trained to generate trajectories for a discrete set of target velocities and turning radius.
A Stochastic Game Framework for Efficient Energy Management in Microgrid Networks
A microgrid is capable of generating a limited amount of energy from a renewable resource and is responsible for handling the demands of its dedicated customers.
Hierarchical Average Reward Policy Gradient Algorithms
Option-critic learning is a general-purpose reinforcement learning (RL) framework that aims to address the issue of long term credit assignment by leveraging temporal abstractions.
A Convergent Off-Policy Temporal Difference Algorithm
In this work, we propose a convergent on-line off-policy TD algorithm under linear function approximation.
Generalized Speedy Q-learning
In most RL algorithms such as Q-learning, the Bellman equation and the Bellman operator play an important role.
A Generalized Minimax Q-learning Algorithm for Two-Player Zero-Sum Stochastic Games
This problem is formulated as a min-max Markov game in the literature.
Learning Active Spine Behaviors for Dynamic and Efficient Locomotion in Quadruped Robots
In this work, we provide a simulation framework to perform systematic studies on the effects of spinal joint compliance and actuation on bounding performance of a 16-DOF quadruped spined robot Stoch 2.
Reinforcement Learning in Non-Stationary Environments
In this paper, we thus consider the problem of developing RL methods that obtain optimal decisions in a non-stationary environment.
Generalized Second Order Value Iteration in Markov Decision Processes
In this work, we propose a second order value iteration procedure that is obtained by applying the Newton-Raphson method to the successive relaxation value iteration scheme.
Successive Over Relaxation Q-Learning
We first derive a modified fixed point iteration for SOR Q-values and utilize stochastic approximation to derive a learning algorithm to compute the optimal value function and an optimal policy.
An Online Sample Based Method for Mode Estimation using ODE Analysis of Stochastic Approximation Algorithms
In many of the practical applications, the analytical form of the density is not known and only the samples from the distribution are available.
Memory-based Deep Reinforcement Learning for Obstacle Avoidance in UAV with Limited Environment Knowledge
When compared to obstacle avoidance in ground vehicular robots, UAV navigation brings in additional challenges because the UAV motion is no more constrained to a well-defined indoor ground or street environment.
Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives
Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs.
Random directions stochastic approximation with deterministic perturbations
We introduce deterministic perturbation schemes for the recently proposed random directions stochastic approximation (RDSA) [17], and propose new first-order and second-order algorithms.
An Online Prediction Algorithm for Reinforcement Learning with Linear Function Approximation using Cross Entropy Method
In this paper, we provide two new stable online algorithms for the problem of prediction in reinforcement learning, \emph{i. e.}, estimating the value function of a model-free Markov reward process using the linear function approximation architecture and with memory and computation costs scaling quadratically in the size of the feature set.
Asynchronous stochastic approximations with asymptotically biased errors and deep multi-agent learning
In this paper, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors.
A Cross Entropy based Optimization Algorithm with Global Convergence Guarantees
The cross entropy (CE) method is a model based search method to solve optimization problems where the objective function has minimal structure.
An Incremental Off-policy Search in a Model-free Markov Decision Process Using a Single Sample Path
In this paper, we consider a modified version of the control problem in a model free Markov decision process (MDP) setting with large state and action spaces.
A unified decision making framework for supply and demand management in microgrid networks
This paper considers two important problems -- on the supply-side and demand-side respectively and studies both in a unified framework.
Analyzing Approximate Value Iteration Algorithms
In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available.
Novel Sensor Scheduling Scheme for Intruder Tracking in Energy Efficient Sensor Networks
We consider the problem of tracking an intruder using a network of wireless sensors.
Multi-Agent Q-Learning for Minimizing Demand-Supply Power Deficit in Microgrids
Finally, we also consider a variant of this problem where the cost of power production at the main site is taken into consideration.
On the function approximation error for risk-sensitive reinforcement learning
The novelty of our approach is that we use the irreduciblity of Markov chain to get the new bounds whereas the earlier work by Basu et al. used spectral variation bound which is true for any matrix.
Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach
Our method, when tested on a 25 node Hadoop cluster shows 66\% decrease in execution time of Hadoop jobs on an average, when compared to the default configuration.
On a convergent off -policy temporal difference learning algorithm in on-line learning environment
In this paper we provide a rigorous convergence analysis of a "off"-policy temporal difference learning algorithm with linear function approximation and per time-step linear computational complexity in "online" learning environment.
Analysis of gradient descent methods with non-diminishing, bounded errors
The main aim of this paper is to provide an analysis of gradient descent (GD) algorithms with gradient errors that do not necessarily vanish, asymptotically.
Shaping Proto-Value Functions via Rewards
In this paper, we combine task-dependent reward shaping and task-independent proto-value functions to obtain reward dependent proto-value functions (RPVFs).
A constrained optimization perspective on actor critic algorithms and application to network routing
We propose a novel actor-critic algorithm with guaranteed convergence to an optimal policy for a discounted reward Markov decision process.
A Study of Gradient Descent Schemes for General-Sum Stochastic Games
However, the optimization problem there has a non-linear objective and non-linear constraints with special structure.
Stability of Stochastic Approximations with `Controlled Markov' Noise and Temporal Difference Learning
Analyzing this class of algorithms is important, since many reinforcement learning (RL) algorithms can be cast as SAs driven by a `controlled Markov' process.
Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learning
We present for the first time an asymptotic convergence analysis of two time-scale stochastic approximation driven by `controlled' Markov noise.
Energy Sharing for Multiple Sensor Nodes with Finite Buffers
We also develop a cross entropy based method that incorporates policy parameterization in order to find near optimal energy sharing policies.
Adaptive system optimization using random directions stochastic approximation
We prove the unbiasedness of both gradient and Hessian estimates and asymptotic (strong) convergence for both first-order and second-order schemes.
A Generalization of the Borkar-Meyn Theorem for Stochastic Recursive Inclusions
In this paper the stability theorem of Borkar and Meyn is extended to include the case when the mean field is a differential inclusion.
Stochastic recursive inclusion in two timescales with an application to the Lagrangian dual problem
In this paper we present a framework to analyze the asymptotic behavior of two timescale stochastic approximation algorithms including those with set-valued mean fields.
Universal Option Models
We prove that the UOM of an option can construct a traditional option model given a reward function, and the option-conditional return is computed directly by a single dot-product of the UOM with the reward function.
Actor-Critic Algorithms for Learning Nash Equilibria in N-player General-Sum Games
We then provide a characterization of solution points of these sub-problems that correspond to Nash equilibria of the underlying game and for this purpose, we derive a set of necessary and sufficient SG-SP (Stochastic Game - Sub-Problem) conditions.
Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks
For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP.
Smoothed Functional Algorithms for Stochastic Optimization using q-Gaussian Distributions
This motivates us to study SF schemes for gradient estimation using the q-Gaussian distribution.
Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation
We introduce the first temporal-difference learning algorithms that converge with smooth value function approximators, such as neural networks.
Multi-Step Dyna Planning for Policy Evaluation and Control
We extend Dyna planning architecture for policy evaluation and control in two significant aspects.
