First return, then explore

27 Apr 2020  ·  Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, Jeff Clune ·

The promise of reinforcement learning is to solve complex sequential decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse and deceptive feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but creating algorithms that can do so remains one of the central challenges of the field. We hypothesise that the main impediment to effective exploration originates from algorithms forgetting how to reach previously visited states ("detachment") and from failing to first return to a state before exploring from it ("derailment"). We introduce Go-Explore, a family of algorithms that addresses these two challenges directly through the simple principles of explicitly remembering promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games, with orders of magnitude improvements on the grand challenges Montezuma's Revenge and Pitfall. We also demonstrate the practical potential of Go-Explore on a sparse-reward pick-and-place robotics task. Additionally, we show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handle stochasticity throughout training. The substantial performance gains from Go-Explore suggest that the simple principles of remembering states, returning to them, and exploring from them are a powerful and general approach to exploration, an insight that may prove critical to the creation of truly intelligent learning agents.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Atari Games Atari 2600 Berzerk Go-Explore Score 197376 # 1
Atari Games Atari 2600 Bowling Go-Explore Score 260 # 2
Atari Games Atari 2600 Centipede Go-Explore Score 1422628 # 1
Atari Games Atari 2600 Freeway Go-Explore Score 34 # 1
Atari Games Atari 2600 Gravitar Go-Explore Score 7588 # 4
Atari Games Atari 2600 Montezuma's Revenge Go-Explore Score 43791 # 1
Atari Games Atari 2600 Pitfall! Go-Explore Score 6954 # 3
Atari Games Atari 2600 Private Eye Go-Explore Score 95756 # 1
Atari Games Atari 2600 Skiing Go-Explore Score -3660 # 3
Atari Games Atari 2600 Solaris Go-Explore Score 19671 # 2
Atari Games Atari 2600 Venture Go-Explore Score 2281 # 2
Atari Games Atari games Go-Explore Mean Human Normalized Score 4989.94% # 4