Asynchronous Methods for Deep Reinforcement Learning

4 Feb 2016Volodymyr MnihAdrià Puigdomènech BadiaMehdi MirzaAlex GravesTimothy P. LillicrapTim HarleyDavid SilverKoray Kavukcuoglu

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers... (read more)

PDF Abstract

Code


Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Atari Games Atari 2600 Alien A3C LSTM hs Score 945.3 # 15
Atari Games Atari 2600 Alien A3C FF hs Score 518.4 # 20
Atari Games Atari 2600 Alien A3C FF (1 day) hs Score 182.1 # 21
Atari Games Atari 2600 Amidar A3C LSTM hs Score 173.0 # 17
Atari Games Atari 2600 Amidar A3C FF hs Score 263.9 # 12
Atari Games Atari 2600 Amidar A3C FF (1 day) hs Score 283.9 # 11
Atari Games Atari 2600 Assault A3C FF (1 day) hs Score 3746.1 # 16
Atari Games Atari 2600 Assault A3C FF hs Score 5474.9 # 11
Atari Games Atari 2600 Assault A3C LSTM hs Score 14497.9 # 2
Atari Games Atari 2600 Asterix A3C FF hs Score 22140.5 # 8
Atari Games Atari 2600 Asterix A3C LSTM hs Score 17244.5 # 12
Atari Games Atari 2600 Asterix A3C FF (1 day) hs Score 6723.0 # 15
Atari Games Atari 2600 Asteroids A3C FF hs Score 4474.5 # 2
Atari Games Atari 2600 Asteroids A3C LSTM hs Score 5093.1 # 1
Atari Games Atari 2600 Asteroids A3C FF (1 day) hs Score 3009.4 # 3
Atari Games Atari 2600 Atlantis A3C FF (1 day) hs Score 772392.0 # 7
Atari Games Atari 2600 Atlantis A3C FF hs Score 911091.0 # 4
Atari Games Atari 2600 Atlantis A3C LSTM hs Score 875822.0 # 5
Atari Games Atari 2600 Bank Heist A3C LSTM hs Score 932.8 # 13
Atari Games Atari 2600 Bank Heist A3C FF hs Score 970.1 # 11
Atari Games Atari 2600 Bank Heist A3C FF (1 day) hs Score 946.0 # 12
Atari Games Atari 2600 Battle Zone A3C LSTM hs Score 20760.0 # 15
Atari Games Atari 2600 Battle Zone A3C FF hs Score 12950.0 # 19
Atari Games Atari 2600 Battle Zone A3C FF (1 day) hs Score 11340.0 # 20
Atari Games Atari 2600 Beam Rider A3C LSTM hs Score 24622.2 # 5
Atari Games Atari 2600 Beam Rider A3C FF hs Score 22707.9 # 8
Atari Games Atari 2600 Beam Rider A3C FF (1 day) hs Score 13235.9 # 13
Atari Games Atari 2600 Berzerk A3C FF (1 day) hs Score 1433.4 # 5
Atari Games Atari 2600 Berzerk A3C FF hs Score 817.9 # 14
Atari Games Atari 2600 Berzerk A3C LSTM hs Score 862.2 # 13
Atari Games Atari 2600 Bowling A3C FF hs Score 35.1 # 20
Atari Games Atari 2600 Bowling A3C LSTM hs Score 41.8 # 17
Atari Games Atari 2600 Bowling A3C FF (1 day) hs Score 36.2 # 19
Atari Games Atari 2600 Boxing A3C LSTM hs Score 37.3 # 20
Atari Games Atari 2600 Boxing A3C FF hs Score 59.8 # 17
Atari Games Atari 2600 Boxing A3C FF (1 day) hs Score 33.7 # 21
Atari Games Atari 2600 Breakout A3C LSTM hs Score 766.8 # 2
Atari Games Atari 2600 Breakout A3C FF hs Score 681.9 # 5
Atari Games Atari 2600 Breakout A3C FF (1 day) hs Score 551.6 # 6
Atari Games Atari 2600 Centipede A3C FF (1 day) hs Score 3306.5 # 21
Atari Games Atari 2600 Centipede A3C FF hs Score 3755.8 # 19
Atari Games Atari 2600 Centipede A3C LSTM hs Score 1997.0 # 22
Atari Games Atari 2600 Chopper Command A3C FF hs Score 7021.0 # 8
Atari Games Atari 2600 Chopper Command A3C LSTM hs Score 10150.0 # 5
Atari Games Atari 2600 Chopper Command A3C FF (1 day) hs Score 4669.0 # 13
Atari Games Atari 2600 Crazy Climber A3C LSTM hs Score 138518.0 # 6
Atari Games Atari 2600 Crazy Climber A3C FF (1 day) hs Score 101624.0 # 17
Atari Games Atari 2600 Crazy Climber A3C FF hs Score 112646.0 # 15
Atari Games Atari 2600 Demon Attack A3C FF hs Score 113308.4 # 4
Atari Games Atari 2600 Demon Attack A3C LSTM hs Score 115201.9 # 3
Atari Games Atari 2600 Demon Attack A3C FF (1 day) hs Score 84997.5 # 5
Atari Games Atari 2600 Double Dunk A3C LSTM hs Score 0.1 # 7
Atari Games Atari 2600 Double Dunk A3C FF hs Score -0.1 # 8
Atari Games Atari 2600 Double Dunk A3C FF (1 day) hs Score 0.1 # 7
Atari Games Atari 2600 Enduro A3C FF (1 day) hs Score -82.2 # 21
Atari Games Atari 2600 Enduro A3C LSTM hs Score -82.5 # 22
Atari Games Atari 2600 Enduro A3C FF hs Score -82.5 # 22
Atari Games Atari 2600 Fishing Derby A3C FF (1 day) hs Score 13.6 # 11
Atari Games Atari 2600 Fishing Derby A3C LSTM hs Score 22.6 # 7
Atari Games Atari 2600 Fishing Derby A3C FF hs Score 18.8 # 8
Atari Games Atari 2600 Freeway A3C LSTM hs Score 0.1 # 22
Atari Games Atari 2600 Freeway A3C FF hs Score 0.1 # 22
Atari Games Atari 2600 Freeway A3C FF (1 day) hs Score 0.1 # 22
Atari Games Atari 2600 Frostbite A3C LSTM hs Score 197.6 # 23
Atari Games Atari 2600 Frostbite A3C FF (1 day) hs Score 180.1 # 26
Atari Games Atari 2600 Frostbite A3C FF hs Score 190.5 # 24
Atari Games Atari 2600 Gopher A3C FF hs Score 10022.8 # 14
Atari Games Atari 2600 Gopher A3C FF (1 day) hs Score 8442.8 # 17
Atari Games Atari 2600 Gopher A3C LSTM hs Score 17106.8 # 10
Atari Games Atari 2600 Gravitar A3C LSTM hs Score 320.0 # 14
Atari Games Atari 2600 Gravitar A3C FF (1 day) hs Score 269.5 # 20
Atari Games Atari 2600 Gravitar A3C FF hs Score 303.5 # 16
Atari Games Atari 2600 HERO A3C FF (1 day) hs Score 28765.8 # 4
Atari Games Atari 2600 HERO A3C FF hs Score 32464.1 # 2
Atari Games Atari 2600 HERO A3C LSTM hs Score 28889.5 # 3
Atari Games Atari 2600 Ice Hockey A3C FF hs Score -2.8 # 12
Atari Games Atari 2600 Ice Hockey A3C LSTM hs Score -1.7 # 8
Atari Games Atari 2600 Ice Hockey A3C FF (1 day) hs Score -4.7 # 16
Atari Games Atari 2600 James Bond A3C LSTM hs Score 613.0 # 12
Atari Games Atari 2600 James Bond A3C FF (1 day) hs Score 351.5 # 20
Atari Games Atari 2600 James Bond A3C FF hs Score 541.0 # 16
Atari Games Atari 2600 Kangaroo A3C FF hs Score 94.0 # 21
Atari Games Atari 2600 Kangaroo A3C LSTM hs Score 125.0 # 19
Atari Games Atari 2600 Kangaroo A3C FF (1 day) hs Score 106.0 # 20
Atari Games Atari 2600 Krull A3C FF (1 day) hs Score 8066.6 # 10
Atari Games Atari 2600 Krull A3C FF hs Score 5560.0 # 19
Atari Games Atari 2600 Krull A3C LSTM hs Score 5911.4 # 18
Atari Games Atari 2600 Kung-Fu Master A3C FF (1 day) hs Score 3046.0 # 21
Atari Games Atari 2600 Kung-Fu Master A3C LSTM hs Score 40835.0 # 4
Atari Games Atari 2600 Kung-Fu Master A3C FF hs Score 28819.0 # 14
Atari Games Atari 2600 Montezuma's Revenge A3C FF (1 day) hs Score 53.0 # 15
Atari Games Atari 2600 Montezuma's Revenge A3C LSTM hs Score 41.0 # 19
Atari Games Atari 2600 Montezuma's Revenge A3C FF hs Score 67.0 # 14
Atari Games Atari 2600 Ms. Pacman A3C LSTM hs Score 850.7 # 19
Atari Games Atari 2600 Ms. Pacman A3C FF (1 day) hs Score 594.4 # 21
Atari Games Atari 2600 Ms. Pacman A3C FF hs Score 653.7 # 20
Atari Games Atari 2600 Name This Game A3C LSTM hs Score 12093.7 # 7
Atari Games Atari 2600 Name This Game A3C FF hs Score 10476.1 # 13
Atari Games Atari 2600 Name This Game A3C FF (1 day) hs Score 5614.0 # 19
Atari Games Atari 2600 Pong A3C FF (1 day) hs Score 11.4 # 11
Atari Games Atari 2600 Pong A3C FF hs Score 5.6 # 13
Atari Games Atari 2600 Pong A3C LSTM hs Score 10.7 # 12
Atari Games Atari 2600 Private Eye A3C FF (1 day) hs Score 194.4 # 17
Atari Games Atari 2600 Private Eye A3C LSTM hs Score 421.1 # 10
Atari Games Atari 2600 Private Eye A3C FF hs Score 206.9 # 14
Atari Games Atari 2600 Q*Bert A3C LSTM hs Score 21307.5 # 3
Atari Games Atari 2600 Q*Bert A3C FF hs Score 15148.8 # 8
Atari Games Atari 2600 Q*Bert A3C FF (1 day) hs Score 13752.3 # 13
Atari Games Atari 2600 River Raid A3C FF (1 day) hs Score 10001.2 # 14
Atari Games Atari 2600 River Raid A3C LSTM hs Score 6591.9 # 17
Atari Games Atari 2600 River Raid A3C FF hs Score 12201.8 # 11
Atari Games Atari 2600 Road Runner A3C FF (1 day) hs Score 31769.0 # 18
Atari Games Atari 2600 Road Runner A3C FF hs Score 34216.0 # 17
Atari Games Atari 2600 Road Runner A3C LSTM hs Score 73949.0 # 1
Atari Games Atari 2600 Robotank A3C LSTM hs Score 2.6 # 21
Atari Games Atari 2600 Robotank A3C FF hs Score 32.8 # 15
Atari Games Atari 2600 Robotank A3C FF (1 day) hs Score 2.3 # 22
Atari Games Atari 2600 Seaquest A3C FF hs Score 2355.4 # 15
Atari Games Atari 2600 Seaquest A3C FF (1 day) hs Score 2300.2 # 16
Atari Games Atari 2600 Seaquest A3C LSTM hs Score 1326.1 # 20
Atari Games Atari 2600 Space Invaders A3C FF (1 day) hs Score 2214.7 # 15
Atari Games Atari 2600 Space Invaders A3C FF hs Score 15730.5 # 3
Atari Games Atari 2600 Space Invaders A3C LSTM hs Score 23846.0 # 2
Atari Games Atari 2600 Star Gunner A3C LSTM hs Score 164766.0 # 1
Atari Games Atari 2600 Star Gunner A3C FF hs Score 138218.0 # 2
Atari Games Atari 2600 Star Gunner A3C FF (1 day) hs Score 64393.0 # 8
Atari Games Atari 2600 Tennis A3C FF hs Score -6.3 # 14
Atari Games Atari 2600 Tennis A3C LSTM hs Score -6.4 # 15
Atari Games Atari 2600 Tennis A3C FF (1 day) hs Score -10.2 # 17
Atari Games Atari 2600 Time Pilot A3C LSTM hs Score 27202.0 # 1
Atari Games Atari 2600 Time Pilot A3C FF hs Score 12679.0 # 2
Atari Games Atari 2600 Time Pilot A3C FF (1 day) hs Score 5825.0 # 15
Atari Games Atari 2600 Tutankham A3C LSTM hs Score 144.2 # 11
Atari Games Atari 2600 Tutankham A3C FF (1 day) hs Score 26.1 # 22
Atari Games Atari 2600 Tutankham A3C FF hs Score 156.3 # 10
Atari Games Atari 2600 Up and Down A3C LSTM hs Score 105728.7 # 1
Atari Games Atari 2600 Up and Down A3C FF hs Score 74705.7 # 3
Atari Games Atari 2600 Up and Down A3C FF (1 day) hs Score 54525.4 # 5
Atari Games Atari 2600 Venture A3C FF hs Score 23.0 # 23
Atari Games Atari 2600 Venture A3C LSTM hs Score 25.0 # 22
Atari Games Atari 2600 Venture A3C FF (1 day) hs Score 19.0 # 25
Atari Games Atari 2600 Video Pinball A3C FF (1 day) hs Score 185852.6 # 13
Atari Games Atari 2600 Video Pinball A3C FF hs Score 331628.1 # 8
Atari Games Atari 2600 Video Pinball A3C LSTM hs Score 470310.5 # 5
Atari Games Atari 2600 Wizard of Wor A3C FF hs Score 17244.0 # 3
Atari Games Atari 2600 Wizard of Wor A3C LSTM hs Score 18082.0 # 2
Atari Games Atari 2600 Wizard of Wor A3C FF (1 day) hs Score 5278.0 # 14
Atari Games Atari 2600 Zaxxon A3C FF (1 day) hs Score 2659.0 # 21
Atari Games Atari 2600 Zaxxon A3C LSTM hs Score 23519.0 # 2
Atari Games Atari 2600 Zaxxon A3C FF hs Score 24622.0 # 1