Deep Exploration via Bootstrapped DQN

Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.

PDF Abstract NeurIPS 2016 PDF NeurIPS 2016 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien Bootstrapped DQN Score 2436.6 # 28
Atari Games Atari 2600 Amidar Bootstrapped DQN Score 1272.5 # 21
Atari Games Atari 2600 Assault Bootstrapped DQN Score 8047.1 # 21
Atari Games Atari 2600 Asterix Bootstrapped DQN Score 19713.2 # 27
Atari Games Atari 2600 Asteroids Bootstrapped DQN Score 1032 # 38
Atari Games Atari 2600 Atlantis Bootstrapped DQN Score 994500 # 11
Atari Games Atari 2600 Bank Heist Bootstrapped DQN Score 1208 # 18
Atari Games Atari 2600 Battle Zone Bootstrapped DQN Score 38666.7 # 17
Atari Games Atari 2600 Beam Rider Bootstrapped DQN Score 23429.8 # 17
Atari Games Atari 2600 Bowling Bootstrapped DQN Score 60.2 # 24
Atari Games Atari 2600 Boxing Bootstrapped DQN Score 93.2 # 27
Atari Games Atari 2600 Breakout Bootstrapped DQN Score 855 # 6
Atari Games Atari 2600 Centipede Bootstrapped DQN Score 4553.5 # 33
Atari Games Atari 2600 Chopper Command Bootstrapped DQN Score 4100 # 35
Atari Games Atari 2600 Crazy Climber Bootstrapped DQN Score 137925.9 # 21
Atari Games Atari 2600 Demon Attack Bootstrapped DQN Score 82610 # 20
Atari Games Atari 2600 Double Dunk Bootstrapped DQN Score 3 # 17
Atari Games Atari 2600 Enduro Bootstrapped DQN Score 1591 # 24
Atari Games Atari 2600 Fishing Derby Bootstrapped DQN Score 26 # 25
Atari Games Atari 2600 Freeway Bootstrapped DQN Score 33.9 # 10
Atari Games Atari 2600 Frostbite Bootstrapped DQN Score 2181.4 # 31
Atari Games Atari 2600 Gopher Bootstrapped DQN Score 17438.4 # 24
Atari Games Atari 2600 Gravitar Bootstrapped DQN Score 286.1 # 46
Atari Games Atari 2600 HERO Bootstrapped DQN Score 21021.3 # 26
Atari Games Atari 2600 Ice Hockey Bootstrapped DQN Score -1.3 # 20
Atari Games Atari 2600 James Bond Bootstrapped DQN Score 1663.5 # 19
Atari Games Atari 2600 Kangaroo Bootstrapped DQN Score 14862.5 # 7
Atari Games Atari 2600 Krull Bootstrapped DQN Score 8627.9 # 26
Atari Games Atari 2600 Kung-Fu Master Bootstrapped DQN Score 36733.3 # 24
Atari Games Atari 2600 Montezuma's Revenge Bootstrapped DQN Score 100 # 26
Atari Games Atari 2600 Ms. Pacman Bootstrapped DQN Score 2983.3 # 28
Atari Games Atari 2600 Name This Game Bootstrapped DQN Score 11501.1 # 25
Atari Games Atari 2600 Pong Bootstrapped DQN Score 20.9 # 17
Atari Games Atari 2600 Private Eye Bootstrapped DQN Score 1812.5 # 20
Atari Games Atari 2600 Q*Bert Bootstrapped DQN Score 15092.7 # 27
Atari Games Atari 2600 River Raid Bootstrapped DQN Score 12845 # 23
Atari Games Atari 2600 Road Runner Bootstrapped DQN Score 51500 # 26
Atari Games Atari 2600 Robotank Bootstrapped DQN Score 66.6 # 12
Atari Games Atari 2600 Seaquest Bootstrapped DQN Score 9083.1 # 23
Atari Games Atari 2600 Space Invaders Bootstrapped DQN Score 2893 # 27
Atari Games Atari 2600 Star Gunner Bootstrapped DQN Score 55725 # 30
Atari Games Atari 2600 Tennis Bootstrapped DQN Score 0 # 20
Atari Games Atari 2600 Time Pilot Bootstrapped DQN Score 9079.4 # 25
Atari Games Atari 2600 Tutankham Bootstrapped DQN Score 214.8 # 22
Atari Games Atari 2600 Up and Down Bootstrapped DQN Score 26231 # 25
Atari Games Atari 2600 Venture Bootstrapped DQN Score 212.5 # 27
Atari Games Atari 2600 Video Pinball Bootstrapped DQN Score 811610 # 9
Atari Games Atari 2600 Wizard of Wor Bootstrapped DQN Score 6804.7 # 28
Atari Games Atari 2600 Zaxxon Bootstrapped DQN Score 11491.7 # 22

Methods