Deep Reinforcement Learning with Double Q-learning

22 Sep 2015  ·  Hado van Hasselt, Arthur Guez, David Silver ·

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien DQN noop Score 1620.0 # 31
Atari Games Atari 2600 Alien Prior+Duel hs Score 823.7 # 42
Atari Games Atari 2600 Alien DDQN (tuned) hs Score 1033.4 # 38
Atari Games Atari 2600 Alien DQN hs Score 634.0 # 45
Atari Games Atari 2600 Amidar Prior+Duel hs Score 238.4 # 34
Atari Games Atari 2600 Amidar DDQN (tuned) hs Score 169.1 # 44
Atari Games Atari 2600 Amidar DQN hs Score 178.4 # 41
Atari Games Atari 2600 Amidar DQN noop Score 978.0 # 26
Atari Games Atari 2600 Assault Prior+Duel hs Score 10950.6 # 17
Atari Games Atari 2600 Assault DQN hs Score 3489.3 # 34
Atari Games Atari 2600 Assault DQN noop Score 4280.4 # 30
Atari Games Atari 2600 Assault DDQN (tuned) hs Score 6060.8 # 25
Atari Games Atari 2600 Asterix DQN noop Score 4359.0 # 40
Atari Games Atari 2600 Asterix DDQN (tuned) hs Score 16837.0 # 34
Atari Games Atari 2600 Asterix Prior+Duel hs Score 364200.0 # 11
Atari Games Atari 2600 Asterix DQN hs Score 3170.5 # 43
Atari Games Atari 2600 Asteroids Prior+Duel hs Score 1021.9 # 39
Atari Games Atari 2600 Asteroids DQN hs Score 1458.7 # 34
Atari Games Atari 2600 Asteroids DQN noop Score 1364.5 # 35
Atari Games Atari 2600 Asteroids DDQN (tuned) hs Score 1193.2 # 36
Atari Games Atari 2600 Atlantis DDQN (tuned) hs Score 319688.0 # 33
Atari Games Atari 2600 Atlantis DQN hs Score 292491.0 # 35
Atari Games Atari 2600 Atlantis DQN noop Score 279987.0 # 36
Atari Games Atari 2600 Atlantis Prior+Duel hs Score 423252.0 # 27
Atari Games Atari 2600 Bank Heist DQN noop Score 455.0 # 35
Atari Games Atari 2600 Bank Heist DQN hs Score 312.7 # 38
Atari Games Atari 2600 Bank Heist Prior+Duel hs Score 1004.6 # 25
Atari Games Atari 2600 Bank Heist DDQN (tuned) hs Score 886.0 # 30
Atari Games Atari 2600 Battle Zone Prior+Duel hs Score 30650.0 # 25
Atari Games Atari 2600 Battle Zone DQN noop Score 29900.0 # 26
Atari Games Atari 2600 Battle Zone DQN hs Score 23750.0 # 34
Atari Games Atari 2600 Battle Zone DDQN (tuned) hs Score 24740.0 # 33
Atari Games Atari 2600 Beam Rider DQN noop Score 8627.5 # 34
Atari Games Atari 2600 Beam Rider Prior+Duel hs Score 37412.2 # 10
Atari Games Atari 2600 Beam Rider DQN hs Score 9743.2 # 33
Atari Games Atari 2600 Beam Rider DDQN (tuned) hs Score 17417.2 # 23
Atari Games Atari 2600 Berzerk DQN noop Score 585.6 # 37
Atari Games Atari 2600 Berzerk Prior+Duel hs Score 2178.6 # 16
Atari Games Atari 2600 Berzerk DDQN (tuned) hs Score 1011.1 # 28
Atari Games Atari 2600 Berzerk DQN hs Score 493.4 # 39
Atari Games Atari 2600 Bowling DQN noop Score 50.4 # 30
Atari Games Atari 2600 Bowling Prior+Duel hs Score 50.4 # 30
Atari Games Atari 2600 Bowling DQN hs Score 56.5 # 27
Atari Games Atari 2600 Bowling DDQN (tuned) hs Score 69.6 # 19
Atari Games Atari 2600 Boxing DQN hs Score 70.3 # 37
Atari Games Atari 2600 Boxing DQN noop Score 88.0 # 30
Atari Games Atari 2600 Boxing Prior+Duel hs Score 79.2 # 31
Atari Games Atari 2600 Boxing DDQN (tuned) hs Score 73.5 # 34
Atari Games Atari 2600 Breakout DQN noop Score 385.5 # 29
Atari Games Atari 2600 Breakout Prior+Duel hs Score 354.6 # 35
Atari Games Atari 2600 Breakout DDQN (tuned) hs Score 368.9 # 32
Atari Games Atari 2600 Breakout DQN hs Score 354.5 # 36
Atari Games Atari 2600 Centipede Prior+Duel hs Score 5570.2 # 28
Atari Games Atari 2600 Centipede DQN noop Score 4657.7 # 31
Atari Games Atari 2600 Centipede DQN hs Score 3973.9 # 37
Atari Games Atari 2600 Centipede DDQN (tuned) hs Score 3853.5 # 39
Atari Games Atari 2600 Chopper Command DQN noop Score 6126.0 # 27
Atari Games Atari 2600 Chopper Command DQN hs Score 5017.0 # 32
Atari Games Atari 2600 Chopper Command Prior+Duel hs Score 8058.0 # 22
Atari Games Atari 2600 Chopper Command DDQN (tuned) hs Score 3495.0 # 39
Atari Games Atari 2600 Crazy Climber DDQN (tuned) hs Score 113782.0 # 34
Atari Games Atari 2600 Crazy Climber DQN noop Score 110763.0 # 36
Atari Games Atari 2600 Crazy Climber DQN hs Score 98128.0 # 39
Atari Games Atari 2600 Crazy Climber Prior+Duel hs Score 127853.0 # 26
Atari Games Atari 2600 Demon Attack DDQN (tuned) hs Score 69803.4 # 26
Atari Games Atari 2600 Demon Attack DQN hs Score 12550.7 # 37
Atari Games Atari 2600 Demon Attack DQN noop Score 12149.4 # 38
Atari Games Atari 2600 Demon Attack Prior+Duel hs Score 73371.3 # 22
Atari Games Atari 2600 Double Dunk DDQN (tuned) hs Score -0.3 # 26
Atari Games Atari 2600 Double Dunk DQN hs Score -6.0 # 26
Atari Games Atari 2600 Double Dunk DQN noop Score -6.6 # 26
Atari Games Atari 2600 Double Dunk Prior+Duel hs Score -10.7 # 26
Atari Games Atari 2600 Enduro DQN hs Score 626.7 # 34
Atari Games Atari 2600 Enduro Prior+Duel hs Score 2223.9 # 14
Atari Games Atari 2600 Enduro DDQN (tuned) hs Score 1216.6 # 27
Atari Games Atari 2600 Enduro DQN noop Score 729.0 # 32
Atari Games Atari 2600 Fishing Derby DQN hs Score -1.6 # 36
Atari Games Atari 2600 Fishing Derby Prior+Duel hs Score 17.0 # 29
Atari Games Atari 2600 Fishing Derby DDQN (tuned) hs Score 3.2 # 35
Atari Games Atari 2600 Fishing Derby DQN noop Score -4.9 # 36
Atari Games Atari 2600 Freeway DQN noop Score 30.8 # 31
Atari Games Atari 2600 Freeway DDQN (tuned) hs Score 28.8 # 38
Atari Games Atari 2600 Freeway Prior+Duel hs Score 28.2 # 39
Atari Games Atari 2600 Freeway DQN hs Score 26.9 # 43
Atari Games Atari 2600 Frostbite Prior+Duel hs Score 4038.4 # 20
Atari Games Atari 2600 Frostbite DQN hs Score 496.1 # 39
Atari Games Atari 2600 Frostbite DDQN (tuned) hs Score 1448.1 # 33
Atari Games Atari 2600 Frostbite DQN noop Score 797.4 # 36
Atari Games Atari 2600 Gopher DDQN (tuned) hs Score 15253.0 # 27
Atari Games Atari 2600 Gopher DQN hs Score 8190.4 # 35
Atari Games Atari 2600 Gopher Prior+Duel hs Score 105148.4 # 10
Atari Games Atari 2600 Gopher DQN noop Score 8777.4 # 32
Atari Games Atari 2600 Gravitar DQN hs Score 298.0 # 44
Atari Games Atari 2600 Gravitar Prior+Duel hs Score 167.0 # 53
Atari Games Atari 2600 Gravitar DQN noop Score 473.0 # 33
Atari Games Atari 2600 Gravitar DDQN (tuned) hs Score 200.5 # 52
Atari Games Atari 2600 HERO Prior+Duel hs Score 15459.2 # 32
Atari Games Atari 2600 HERO DQN noop Score 20437.8 # 29
Atari Games Atari 2600 HERO DQN hs Score 14992.9 # 34
Atari Games Atari 2600 HERO DDQN (tuned) hs Score 14892.5 # 35
Atari Games Atari 2600 Ice Hockey DQN noop Score -1.9 # 20
Atari Games Atari 2600 Ice Hockey DDQN (tuned) hs Score -2.5 # 20
Atari Games Atari 2600 Ice Hockey DQN hs Score -1.6 # 20
Atari Games Atari 2600 Ice Hockey Prior+Duel hs Score 0.5 # 17
Atari Games Atari 2600 James Bond DQN hs Score 697.5 # 29
Atari Games Atari 2600 James Bond DDQN (tuned) hs Score 573.0 # 34
Atari Games Atari 2600 James Bond Prior+Duel hs Score 585.0 # 32
Atari Games Atari 2600 James Bond DQN noop Score 768.5 # 28
Atari Games Atari 2600 Kangaroo DQN noop Score 7259.0 # 26
Atari Games Atari 2600 Kangaroo DQN hs Score 4496.0 # 29
Atari Games Atari 2600 Kangaroo DDQN (tuned) hs Score 11204.0 # 22
Atari Games Atari 2600 Kangaroo Prior+Duel hs Score 861.0 # 41
Atari Games Atari 2600 Krull DQN noop Score 8422.3 # 27
Atari Games Atari 2600 Krull Prior+Duel hs Score 7658.6 # 33
Atari Games Atari 2600 Krull DQN hs Score 6206.0 # 37
Atari Games Atari 2600 Krull DDQN (tuned) hs Score 6796.1 # 35
Atari Games Atari 2600 Kung-Fu Master DQN hs Score 20882.0 # 39
Atari Games Atari 2600 Kung-Fu Master DQN noop Score 26059.0 # 36
Atari Games Atari 2600 Kung-Fu Master Prior+Duel hs Score 37484.0 # 23
Atari Games Atari 2600 Kung-Fu Master DDQN (tuned) hs Score 30207.0 # 32
Atari Games Atari 2600 Montezuma's Revenge DQN hs Score 47.0 # 34
Atari Games Atari 2600 Montezuma's Revenge Prior+Duel hs Score 24.0 # 37
Atari Games Atari 2600 Montezuma's Revenge DDQN (tuned) hs Score 42.0 # 35
Atari Games Atari 2600 Ms. Pacman DQN noop Score 3085.6 # 27
Atari Games Atari 2600 Ms. Pacman DQN hs Score 1092.3 # 41
Atari Games Atari 2600 Ms. Pacman Prior+Duel hs Score 1007.8 # 42
Atari Games Atari 2600 Ms. Pacman DDQN (tuned) hs Score 1241.3 # 39
Atari Games Atari 2600 Name This Game DQN hs Score 6738.8 # 36
Atari Games Atari 2600 Name This Game Prior+Duel hs Score 13637.9 # 19
Atari Games Atari 2600 Name This Game DQN noop Score 8207.8 # 34
Atari Games Atari 2600 Name This Game DDQN (tuned) hs Score 8960.3 # 33
Atari Games Atari 2600 Pong DDQN (tuned) hs Score 19.1 # 35
Atari Games Atari 2600 Pong DQN noop Score 19.5 # 34
Atari Games Atari 2600 Pong DQN hs Score 18.0 # 42
Atari Games Atari 2600 Pong Prior+Duel hs Score 18.4 # 39
Atari Games Atari 2600 Private Eye DQN noop Score 146.7 # 39
Atari Games Atari 2600 Private Eye DDQN (tuned) hs Score -575.5 # 52
Atari Games Atari 2600 Private Eye Prior+Duel hs Score 1277.6 # 22
Atari Games Atari 2600 Private Eye DQN hs Score 207.9 # 32
Atari Games Atari 2600 Q*Bert DQN noop Score 13117.3 # 37
Atari Games Atari 2600 Q*Bert DDQN (tuned) hs Score 11020.8 # 38
Atari Games Atari 2600 Q*Bert Prior+Duel hs Score 14063.0 # 35
Atari Games Atari 2600 Q*Bert DQN hs Score 9271.5 # 41
Atari Games Atari 2600 River Raid Prior+Duel hs Score 16496.8 # 18
Atari Games Atari 2600 River Raid DDQN (tuned) hs Score 10838.4 # 27
Atari Games Atari 2600 River Raid DQN noop Score 7377.6 # 32
Atari Games Atari 2600 River Raid DQN hs Score 4748.5 # 36
Atari Games Atari 2600 Road Runner DQN hs Score 35215.0 # 34
Atari Games Atari 2600 Road Runner DDQN (tuned) hs Score 43156.0 # 30
Atari Games Atari 2600 Road Runner DQN noop Score 39544.0 # 32
Atari Games Atari 2600 Road Runner Prior+Duel hs Score 54630.0 # 23
Atari Games Atari 2600 Robotank DQN hs Score 58.7 # 26
Atari Games Atari 2600 Robotank DQN noop Score 63.9 # 19
Atari Games Atari 2600 Robotank Prior+Duel hs Score 24.7 # 34
Atari Games Atari 2600 Robotank DDQN (tuned) hs Score 59.1 # 25
Atari Games Atari 2600 Seaquest DDQN (tuned) hs Score 14498.0 # 19
Atari Games Atari 2600 Seaquest DQN noop Score 5860.6 # 30
Atari Games Atari 2600 Seaquest DQN hs Score 4216.7 # 34
Atari Games Atari 2600 Seaquest Prior+Duel hs Score 1431.2 # 44
Atari Games Atari 2600 Space Invaders Prior+Duel hs Score 8978.0 # 17
Atari Games Atari 2600 Space Invaders DDQN (tuned) hs Score 2628.7 # 31
Atari Games Atari 2600 Space Invaders DQN noop Score 1692.3 # 38
Atari Games Atari 2600 Space Invaders DQN hs Score 1293.8 # 40
Atari Games Atari 2600 Star Gunner Prior+Duel hs Score 127073.0 # 14
Atari Games Atari 2600 Star Gunner DDQN (tuned) hs Score 58365.0 # 28
Atari Games Atari 2600 Star Gunner DQN hs Score 52970.0 # 32
Atari Games Atari 2600 Star Gunner DQN noop Score 54282.0 # 31
Atari Games Atari 2600 Tennis DDQN (tuned) hs Score -7.8 # 29
Atari Games Atari 2600 Tennis Prior+Duel hs Score -13.2 # 29
Atari Games Atari 2600 Tennis DQN noop Score 12.2 # 13
Atari Games Atari 2600 Tennis DQN hs Score 11.1 # 15
Atari Games Atari 2600 Time Pilot Prior+Duel hs Score 4871.0 # 37
Atari Games Atari 2600 Time Pilot DQN noop Score 4870.0 # 38
Atari Games Atari 2600 Time Pilot DQN hs Score 4786.0 # 40
Atari Games Atari 2600 Time Pilot DDQN (tuned) hs Score 6608.0 # 31
Atari Games Atari 2600 Tutankham DQN hs Score 45.6 # 42
Atari Games Atari 2600 Tutankham DDQN (tuned) hs Score 92.2 # 38
Atari Games Atari 2600 Tutankham DQN noop Score 68.1 # 39
Atari Games Atari 2600 Tutankham Prior+Duel hs Score 108.6 # 36
Atari Games Atari 2600 Up and Down DDQN (tuned) hs Score 19086.9 # 31
Atari Games Atari 2600 Up and Down Prior+Duel hs Score 22681.3 # 29
Atari Games Atari 2600 Up and Down DQN hs Score 8038.5 # 40
Atari Games Atari 2600 Up and Down DQN noop Score 9989.9 # 37
Atari Games Atari 2600 Venture DQN noop Score 163.0 # 30
Atari Games Atari 2600 Venture DQN hs Score 136.0 # 31
Atari Games Atari 2600 Venture Prior+Duel hs Score 29.0 # 41
Atari Games Atari 2600 Venture DDQN (tuned) hs Score 21.0 # 44
Atari Games Atari 2600 Video Pinball DQN noop Score 196760.4 # 27
Atari Games Atari 2600 Video Pinball DQN hs Score 154414.1 # 29
Atari Games Atari 2600 Video Pinball DDQN (tuned) hs Score 367823.7 # 21
Atari Games Atari 2600 Video Pinball Prior+Duel hs Score 447408.6 # 20
Atari Games Atari 2600 Wizard of Wor DQN hs Score 1609.0 # 39
Atari Games Atari 2600 Wizard of Wor DDQN (tuned) hs Score 6201.0 # 29
Atari Games Atari 2600 Wizard of Wor Prior+Duel hs Score 10471.0 # 18
Atari Games Atari 2600 Wizard of Wor DQN noop Score 2704.0 # 37
Atari Games Atari 2600 Zaxxon DQN noop Score 5363.0 # 35
Atari Games Atari 2600 Zaxxon DDQN (tuned) hs Score 8593.0 # 32
Atari Games Atari 2600 Zaxxon Prior+Duel hs Score 11320.0 # 23
Atari Games Atari 2600 Zaxxon DQN hs Score 4412.0 # 37

Methods