IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.

PDF Abstract ICML 2018 PDF ICML 2018 Abstract

Results from the Paper


Ranked #3 on Atari Games on Atari 2600 Skiing (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Atari Games Atari 2600 Alien IMPALA (deep) Score 15962.10 # 10
Atari Games Atari 2600 Amidar IMPALA (deep) Score 1554.79 # 17
Atari Games Atari 2600 Assault IMPALA (deep) Score 19148.47 # 11
Atari Games Atari 2600 Asterix IMPALA (deep) Score 300732.00 # 16
Atari Games Atari 2600 Asteroids IMPALA (deep) Score 108590.05 # 9
Atari Games Atari 2600 Atlantis IMPALA (deep) Score 849967.50 # 21
Atari Games Atari 2600 Bank Heist IMPALA (deep) Score 1223.15 # 16
Atari Games Atari 2600 Battle Zone IMPALA (deep) Score 20885.00 # 36
Atari Games Atari 2600 Beam Rider IMPALA (deep) Score 32463.47 # 12
Atari Games Atari 2600 Berzerk IMPALA (deep) Score 1852.70 # 18
Atari Games Atari 2600 Bowling IMPALA (deep) Score 59.92 # 25
Atari Games Atari 2600 Boxing IMPALA (deep) Score 99.96 # 10
Atari Games Atari 2600 Breakout IMPALA (deep) Score 787.34 # 10
Atari Games Atari 2600 Centipede IMPALA (deep) Score 11049.75 # 18
Atari Games Atari 2600 Chopper Command IMPALA (deep) Score 28255.00 # 12
Atari Games Atari 2600 Crazy Climber IMPALA (deep) Score 136950.00 # 22
Atari Games Atari 2600 Defender IMPALA (deep) Score 185203.00 # 11
Atari Games Atari 2600 Demon Attack IMPALA (deep) Score 132826.98 # 10
Atari Games Atari 2600 Double Dunk IMPALA (deep) Score -0.33 # 26
Atari Games Atari 2600 Enduro IMPALA (deep) Score 0.00 # 45
Atari Games Atari 2600 Fishing Derby IMPALA (deep) Score 44.85 # 15
Atari Games Atari 2600 Freeway IMPALA (deep) Score 0.00 # 57
Atari Games Atari 2600 Frostbite IMPALA (deep) Score 317.75 # 44
Atari Games Atari 2600 Gopher IMPALA (deep) Score 66782.30 # 15
Atari Games Atari 2600 Gravitar IMPALA (deep) Score 359.50 # 40
Atari Games Atari 2600 HERO IMPALA (deep) Score 33730.55 # 9
Atari Games Atari 2600 Ice Hockey IMPALA (deep) Score 3.48 # 14
Atari Games Atari 2600 James Bond IMPALA (deep) Score 601.50 # 31
Atari Games Atari 2600 Kangaroo IMPALA (deep) Score 1632.00 # 35
Atari Games Atari 2600 Krull IMPALA (deep) Score 8147.40 # 28
Atari Games Atari 2600 Kung-Fu Master IMPALA (deep) Score 43375.50 # 19
Atari Games Atari 2600 Montezuma's Revenge IMPALA (deep) Score 0.00 # 42
Atari Games Atari 2600 Ms. Pacman IMPALA (deep) Score 7342.32 # 12
Atari Games Atari 2600 Name This Game IMPALA (deep) Score 21537.20 # 11
Atari Games Atari 2600 Phoenix IMPALA (deep) Score 210996.45 # 10
Atari Games Atari 2600 Pitfall! IMPALA (deep) Score -1.66 # 20
Atari Games Atari 2600 Pong IMPALA (deep) Score 20.98 # 15
Atari Games Atari 2600 Private Eye IMPALA (deep) Score 98.50 # 48
Atari Games Atari 2600 Q*Bert IMPALA (deep) Score 351200.12 # 4
Atari Games Atari 2600 River Raid IMPALA (deep) Score 29608.05 # 6
Atari Games Atari 2600 Road Runner IMPALA (deep) Score 57121.00 # 19
Atari Games Atari 2600 Robotank IMPALA (deep) Score 12.96 # 36
Atari Games Atari 2600 Seaquest IMPALA (deep) Score 1753.20 # 41
Atari Games Atari 2600 Skiing IMPALA (deep) Score -10180.38 # 3
Atari Games Atari 2600 Solaris IMPALA (deep) Score 2365.00 # 19
Atari Games Atari 2600 Space Invaders IMPALA (deep) Score 43595.78 # 9
Atari Games Atari 2600 Star Gunner IMPALA (deep) Score 200625.00 # 8
Atari Games Atari 2600 Surround IMPALA (deep) Score 7.56 # 8
Atari Games Atari 2600 Tennis IMPALA (deep) Score 0.55 # 19
Atari Games Atari 2600 Time Pilot IMPALA (deep) Score 48481.50 # 10
Atari Games Atari 2600 Tutankham IMPALA (deep) Score 292.11 # 11
Atari Games Atari 2600 Up and Down IMPALA (deep) Score 332546.75 # 11
Atari Games Atari 2600 Venture IMPALA (deep) Score 0.00 # 49
Atari Games Atari 2600 Video Pinball IMPALA (deep) Score 572898.27 # 13
Atari Games Atari 2600 Wizard of Wor IMPALA (deep) Score 9157.50 # 22
Atari Games Atari 2600 Yars Revenge IMPALA (deep) Score 84231.14 # 11
Atari Games Atari 2600 Zaxxon IMPALA (deep) Score 32935.50 # 9
Atari Games Atari-57 IMPALA, deep Human World Record Breakthrough 3 # 9
Mean Human Normalized Score 957.34% # 8
Atari Games Atari games IMPALA, deep Mean Human Normalized Score 957.34% # 9

Methods