Asynchronous Methods for Deep Reinforcement Learning

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK BENCHMARK
Atari Games Atari 2600 Alien A3C LSTM hs Score 945.3 # 33
Atari Games Atari 2600 Alien A3C FF (1 day) hs Score 182.1 # 40
Atari Games Atari 2600 Alien A3C FF hs Score 518.4 # 38
Atari Games Atari 2600 Amidar A3C FF (1 day) hs Score 283.9 # 26
Atari Games Atari 2600 Amidar A3C LSTM hs Score 173.0 # 35
Atari Games Atari 2600 Amidar A3C FF hs Score 263.9 # 27
Atari Games Atari 2600 Assault A3C LSTM hs Score 14497.9 # 8
Atari Games Atari 2600 Assault A3C FF (1 day) hs Score 3746.1 # 26
Atari Games Atari 2600 Assault A3C FF hs Score 5474.9 # 20
Atari Games Atari 2600 Asterix A3C FF hs Score 22140.5 # 19
Atari Games Atari 2600 Asterix A3C FF (1 day) hs Score 6723 # 30
Atari Games Atari 2600 Asterix A3C LSTM hs Score 17244.5 # 26
Atari Games Atari 2600 Asteroids A3C FF (1 day) hs Score 3009.4 # 14
Atari Games Atari 2600 Asteroids A3C FF hs Score 4474.5 # 11
Atari Games Atari 2600 Asteroids A3C LSTM hs Score 5093.1 # 8
Atari Games Atari 2600 Atlantis A3C FF (1 day) hs Score 772392.0 # 18
Atari Games Atari 2600 Atlantis A3C LSTM hs Score 875822.0 # 15
Atari Games Atari 2600 Atlantis A3C FF hs Score 911091.0 # 14
Atari Games Atari 2600 Bank Heist A3C FF (1 day) hs Score 946.0 # 23
Atari Games Atari 2600 Bank Heist A3C FF hs Score 970.1 # 22
Atari Games Atari 2600 Bank Heist A3C LSTM hs Score 932.8 # 24
Atari Games Atari 2600 Battle Zone A3C LSTM hs Score 20760.0 # 30
Atari Games Atari 2600 Battle Zone A3C FF (1 day) hs Score 11340.0 # 36
Atari Games Atari 2600 Battle Zone A3C FF hs Score 12950.0 # 35
Atari Games Atari 2600 Beam Rider A3C FF hs Score 22707.9 # 15
Atari Games Atari 2600 Beam Rider A3C FF (1 day) hs Score 13235.9 # 21
Atari Games Atari 2600 Beam Rider A3C LSTM hs Score 24622.2 # 11
Atari Games Atari 2600 Berzerk A3C LSTM hs Score 862.2 # 25
Atari Games Atari 2600 Berzerk A3C FF hs Score 817.9 # 26
Atari Games Atari 2600 Berzerk A3C FF (1 day) hs Score 1433.4 # 15
Atari Games Atari 2600 Bowling A3C FF (1 day) hs Score 36.2 # 32
Atari Games Atari 2600 Bowling A3C FF hs Score 35.1 # 33
Atari Games Atari 2600 Bowling A3C LSTM hs Score 41.8 # 29
Atari Games Atari 2600 Boxing A3C LSTM hs Score 37.3 # 31
Atari Games Atari 2600 Boxing A3C FF (1 day) hs Score 33.7 # 32
Atari Games Atari 2600 Boxing A3C FF hs Score 59.8 # 27
Atari Games Atari 2600 Breakout A3C FF hs Score 681.9 # 12
Atari Games Atari 2600 Breakout A3C FF (1 day) hs Score 551.6 # 13
Atari Games Atari 2600 Breakout A3C LSTM hs Score 766.8 # 8
Atari Games Atari 2600 Centipede A3C FF (1 day) hs Score 3306.5 # 37
Atari Games Atari 2600 Centipede A3C FF hs Score 3755.8 # 33
Atari Games Atari 2600 Centipede A3C LSTM hs Score 1997.0 # 38
Atari Games Atari 2600 Chopper Command A3C LSTM hs Score 10150.0 # 15
Atari Games Atari 2600 Chopper Command A3C FF (1 day) hs Score 4669.0 # 27
Atari Games Atari 2600 Chopper Command A3C FF hs Score 7021.0 # 18
Atari Games Atari 2600 Crazy Climber A3C FF (1 day) hs Score 101624.0 # 31
Atari Games Atari 2600 Crazy Climber A3C LSTM hs Score 138518.0 # 14
Atari Games Atari 2600 Crazy Climber A3C FF hs Score 112646.0 # 29
Atari Games Atari 2600 Demon Attack A3C LSTM hs Score 115201.9 # 10
Atari Games Atari 2600 Demon Attack A3C FF (1 day) hs Score 84997.5 # 13
Atari Games Atari 2600 Demon Attack A3C FF hs Score 113308.4 # 12
Atari Games Atari 2600 Double Dunk A3C LSTM hs Score 0.1 # 18
Atari Games Atari 2600 Double Dunk A3C FF hs Score -0.1 # 19
Atari Games Atari 2600 Double Dunk A3C FF (1 day) hs Score 0.1 # 18
Atari Games Atari 2600 Enduro A3C LSTM hs Score -82.5 # 39
Atari Games