Human level control through deep reinforcement learning

25 Feb 2015  ·  Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg1 & Demis Hassabis ·

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

PDF Abstract

Results from the Paper
Add Remove

Ranked #20 on Atari Games on Atari 2600 Ice Hockey

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Atari Games Atari 2600 Alien Nature DQN Score 3069.0 # 27
Compare
Atari Games Atari 2600 Amidar Nature DQN Score 739.5 # 28
Compare
Atari Games Atari 2600 Assault Nature DQN Score 3359.0 # 35
Compare
Atari Games Atari 2600 Asterix Nature DQN Score 6012 # 39
Compare
Atari Games Atari 2600 Asteroids Nature DQN Score 1629.0 # 31
Compare
Atari Games Atari 2600 Atlantis Nature DQN Score 85641.0 # 40
Compare
Atari Games Atari 2600 Bank Heist Nature DQN Score 429.7 # 36
Compare
Atari Games Atari 2600 Battle Zone Nature DQN Score 26300.0 # 29
Compare
Atari Games Atari 2600 Beam Rider Nature DQN Score 6846.0 # 36
Compare
Atari Games Atari 2600 Bowling Nature DQN Score 42.4 # 36
Compare
Atari Games Atari 2600 Boxing Nature DQN Score 71.8 # 36
Compare
Atari Games Atari 2600 Breakout Nature DQN Score 401.2 # 27
Compare
Atari Games Atari 2600 Centipede Nature DQN Score 8309.0 # 21
Compare
Atari Games Atari 2600 Chopper Command Nature DQN Score 6687.0 # 25
Compare
Atari Games Atari 2600 Crazy Climber Nature DQN Score 114103.0 # 33
Compare
Atari Games Atari 2600 Demon Attack Nature DQN Score 9711.0 # 40
Compare
Atari Games Atari 2600 Double Dunk Nature DQN Score -18.1 # 26
Compare
Atari Games Atari 2600 Enduro Nature DQN Score 301.8 # 37
Compare
Atari Games Atari 2600 Fishing Derby Nature DQN Score -0.8 # 36
Compare
Atari Games Atari 2600 Freeway Nature DQN Score 30.3 # 33
Compare
Atari Games Atari 2600 Frostbite Nature DQN Score 328.3 # 42
Compare
Atari Games Atari 2600 Gopher Nature DQN Score 8520.0 # 33
Compare
Atari Games Atari 2600 Gravitar Nature DQN Score 306.7 # 42
Compare
Atari Games Atari 2600 HERO Nature DQN Score 19950 # 31
Compare
Atari Games Atari 2600 Ice Hockey Nature DQN Score -1.6 # 20
Compare
Atari Games Atari 2600 James Bond Nature DQN Score 576.7 # 33
Compare
Atari Games Atari 2600 Kangaroo Nature DQN Score 6740.0 # 27
Compare
Atari Games Atari 2600 Krull Nature DQN Score 3805.0 # 42
Compare
Atari Games Atari 2600 Kung-Fu Master Nature DQN Score 23270.0 # 38
Compare
Atari Games Atari 2600 Montezuma's Revenge Nature DQN Score 0 # 42
Compare
Atari Games Atari 2600 Ms. Pacman Nature DQN Score 2311.0 # 33
Compare
Atari Games Atari 2600 Name This Game Nature DQN Score 7257.0 # 35
Compare
Atari Games Atari 2600 Pong Nature DQN Score 18.9 # 36
Compare
Atari Games Atari 2600 Private Eye Nature DQN Score 1788.0 # 21
Compare
Atari Games Atari 2600 Q*Bert Nature DQN Score 10596 # 39
Compare
Atari Games Atari 2600 River Raid Nature DQN Score 8316.0 # 30
Compare
Atari Games Atari 2600 Road Runner Nature DQN Score 18257.0 # 37
Compare
Atari Games Atari 2600 Robotank Nature DQN Score 51.6 # 29
Compare
Atari Games Atari 2600 Seaquest Nature DQN Score 5286.0 # 32
Compare
Atari Games Atari 2600 Space Invaders Nature DQN Score 1976.0 # 38
Compare
Atari Games Atari 2600 Star Gunner Nature DQN Score 57997.0 # 29
Compare
Atari Games Atari 2600 Star Gunner SARSA Score 9.4 # 43
Compare
Atari Games Atari 2600 Tennis Nature DQN Score -2.5 # 29
Compare
Atari Games Atari 2600 Time Pilot Nature DQN Score 5947.0 # 34
Compare
Atari Games Atari 2600 Tutankham Nature DQN Score 186.7 # 26
Compare
Atari Games Atari 2600 Up and Down Nature DQN Score 8456.0 # 39
Compare
Atari Games Atari 2600 Venture Nature DQN Score 380.0 # 25
Compare
Atari Games Atari 2600 Video Pinball Nature DQN Score 42684.0 # 36
Compare
Atari Games Atari 2600 Wizard of Wor Nature DQN Score 3393.0 # 36
Compare
Atari Games Atari 2600 Zaxxon Nature DQN Score 4977.0 # 36
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy