Addressing Function Approximation Error in Actor-Critic Methods

ICML 2018  ·  Scott Fujimoto, Herke van Hoof, David Meger ·

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

PDF Abstract ICML 2018 PDF ICML 2018 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
OpenAI Gym Ant-v4 TD3 Average Return 5942.55 # 2
OpenAI Gym HalfCheetah-v4 TD3 Average Return 12026.73 # 3
OpenAI Gym Hopper-v4 TD3 Average Return 3319.98 # 2
OpenAI Gym Humanoid-v4 TD3 Average Return 198.44 # 4