Search Results for author:
Found 8 papers, 1 papers with code
The Role of Baselines in Policy Gradient Optimization
Instead, the analysis reveals that the primary effect of the value baseline is to \textbf{reduce the aggressiveness of the updates} rather than their variance.
Offline-Online Reinforcement Learning: Extending Batch and Online RL
To enable better performance, we investigate the offline-online setting: The agent has access to a batch of data to train on but is also allowed to learn during the evaluation phase in an online manner.
Beyond variance reduction: Understanding the true impact of baselines on policy optimization
Traditionally, stochastic optimization theory predicts that learning dynamics are governed by the curvature of the loss function and the noise of the gradient estimates.
Incrementally Learning Functions of the Return
Temporal difference methods enable efficient estimation of value functions in reinforcement learning in an incremental fashion, and are of broader interest because they correspond learning as observed in biological systems.
Importance Resampling for Off-policy Prediction
Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning.
Two-Timescale Networks for Nonlinear Value Function Approximation
A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control.
Importance Resampling for Off-policy Policy Evaluation
We propose Importance Resampling (IR) for off-policy learning, that resamples experience from the replay buffer and applies a standard on-policy update.
High-confidence error estimates for learned value functions
We provide experiments investigating the number of samples required by this offline algorithm in simple benchmark reinforcement learning domains, and highlight that there are still many open questions to be solved for this important problem.
