The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Atari Games Atari 2600 Alien Nature DQN Score 3069.0 # 27
Atari Games Atari 2600 Amidar Nature DQN Score 739.5 # 28
Atari Games Atari 2600 Assault Nature DQN Score 3359.0 # 35
Atari Games Atari 2600 Asterix Nature DQN Score 6012 # 39
Atari Games Atari 2600 Asteroids Nature DQN Score 1629.0 # 31
Atari Games Atari 2600 Atlantis Nature DQN Score 85641.0 # 40
Atari Games Atari 2600 Bank Heist Nature DQN Score 429.7 # 36
Atari Games Atari 2600 Battle Zone Nature DQN Score 26300.0 # 29
Atari Games Atari 2600 Beam Rider Nature DQN Score 6846.0 # 36
Atari Games Atari 2600 Bowling Nature DQN Score 42.4 # 36
Atari Games Atari 2600 Boxing Nature DQN Score 71.8 # 36
Atari Games Atari 2600 Breakout Nature DQN Score 401.2 # 27
Atari Games Atari 2600 Centipede Nature DQN Score 8309.0 # 21
Atari Games Atari 2600 Chopper Command Nature DQN Score 6687.0 # 25
Atari Games Atari 2600 Crazy Climber Nature DQN Score 114103.0 # 33
Atari Games Atari 2600 Demon Attack Nature DQN Score 9711.0 # 40
Atari Games Atari 2600 Double Dunk Nature DQN Score -18.1 # 26
Atari Games Atari 2600 Enduro Nature DQN Score 301.8 # 37
Atari Games Atari 2600 Fishing Derby Nature DQN Score -0.8 # 36
Atari Games Atari 2600 Freeway Nature DQN Score 30.3 # 33
Atari Games Atari 2600 Frostbite Nature DQN Score 328.3 # 42
Atari Games Atari 2600 Gopher Nature DQN Score 8520.0 # 33
Atari Games Atari 2600 Gravitar Nature DQN Score 306.7 # 42
Atari Games Atari 2600 HERO Nature DQN Score 19950 # 31
Atari Games Atari 2600 Ice Hockey Nature DQN Score -1.6 # 20
Atari Games Atari 2600 James Bond Nature DQN Score 576.7 # 33
Atari Games Atari 2600 Kangaroo Nature DQN Score 6740.0 # 27
Atari Games Atari 2600 Krull Nature DQN Score 3805.0 # 42
Atari Games Atari 2600 Kung-Fu Master Nature DQN Score 23270.0 # 38
Atari Games Atari 2600 Montezuma's Revenge Nature DQN Score 0 # 42
Atari Games Atari 2600 Ms. Pacman Nature DQN Score 2311.0 # 33
Atari Games Atari 2600 Name This Game Nature DQN Score 7257.0 # 35
Atari Games Atari 2600 Pong Nature DQN Score 18.9 # 36
Atari Games Atari 2600 Private Eye Nature DQN Score 1788.0 # 21
Atari Games Atari 2600 Q*Bert Nature DQN Score 10596 # 39
Atari Games Atari 2600 River Raid Nature DQN Score 8316.0 # 30
Atari Games Atari 2600 Road Runner Nature DQN Score 18257.0 # 37
Atari Games Atari 2600 Robotank Nature DQN Score 51.6 # 29
Atari Games Atari 2600 Seaquest Nature DQN Score 5286.0 # 32
Atari Games Atari 2600 Space Invaders Nature DQN Score 1976.0 # 38
Atari Games Atari 2600 Star Gunner Nature DQN Score 57997.0 # 29
Atari Games Atari 2600 Star Gunner SARSA Score 9.4 # 43
Atari Games Atari 2600 Tennis Nature DQN Score -2.5 # 29
Atari Games Atari 2600 Time Pilot Nature DQN Score 5947.0 # 34
Atari Games Atari 2600 Tutankham Nature DQN Score 186.7 # 26
Atari Games Atari 2600 Up and Down Nature DQN Score 8456.0 # 39
Atari Games Atari 2600 Venture Nature DQN Score 380.0 # 25
Atari Games Atari 2600 Video Pinball Nature DQN Score 42684.0 # 36
Atari Games Atari 2600 Wizard of Wor Nature DQN Score 3393.0 # 36
Atari Games Atari 2600 Zaxxon Nature DQN Score 4977.0 # 36

Methods


No methods listed for this paper. Add relevant methods here