Distributed Prioritized Experience Replay

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

PDF Abstract ICLR 2018 PDF ICLR 2018 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Atari Games Atari 2600 Alien Ape-X Score 40804.9 # 8
Atari Games Atari 2600 Amidar Ape-X Score 8659.2 # 4
Atari Games Atari 2600 Assault Ape-X Score 24559.4 # 8
Atari Games Atari 2600 Asterix Ape-X Score 313305 # 15
Atari Games Atari 2600 Asteroids Ape-X Score 155495.1 # 7
Atari Games Atari 2600 Atlantis Ape-X Score 944497.5 # 17
Atari Games Atari 2600 Bank Heist Ape-X Score 1716.4 # 4
Atari Games Atari 2600 Battle Zone Ape-X Score 98895 # 7
Atari Games Atari 2600 Beam Rider Ape-X Score 63305.2 # 8
Atari Games Atari 2600 Berzerk Ape-X Score 57196.7 # 4
Atari Games Atari 2600 Bowling Ape-X Score 17.6 # 44
Atari Games Atari 2600 Boxing Ape-X Score 100 # 1
Atari Games Atari 2600 Breakout Ape-X Score 800.9 # 8
Atari Games Atari 2600 Centipede Ape-X Score 12974 # 14
Atari Games Atari 2600 Chopper Command Ape-X Score 721851 # 8
Atari Games Atari 2600 Crazy Climber Ape-X Score 320426 # 4
Atari Games Atari 2600 Defender Ape-X Score 411943.5 # 9
Atari Games Atari 2600 Demon Attack Ape-X Score 133086.4 # 9
Atari Games Atari 2600 Double Dunk Ape-X Score 23.5 # 9
Atari Games Atari 2600 Enduro Ape-X Score 2177.4 # 15
Atari Games Atari 2600 Fishing Derby Ape-X Score 44.4 # 16
Atari Games Atari 2600 Freeway Ape-X Score 33.7 # 14
Atari Games Atari 2600 Frostbite Ape-X Score 9328.6 # 11
Atari Games Atari 2600 Gopher Ape-X Score 120500.9 # 6
Atari Games Atari 2600 Gravitar Ape-X Score 1598.5 # 19
Atari Games Atari 2600 HERO Ape-X Score 31655.9 # 12
Atari Games Atari 2600 Ice Hockey Ape-X Score 33 # 9
Atari Games Atari 2600 James Bond Ape-X Score 21322.5 # 11
Atari Games Atari 2600 Kangaroo Ape-X Score 1416 # 38
Atari Games Atari 2600 Krull Ape-X Score 11741.4 # 10
Atari Games Atari 2600 Kung-Fu Master Ape-X Score 97829.5 # 10
Atari Games Atari 2600 Montezuma's Revenge Ape-X Score 2500.0 # 15
Atari Games Atari 2600 Ms. Pacman Ape-X Score 11255.2 # 9
Atari Games Atari 2600 Name This Game Ape-X Score 25783.3 # 8
Atari Games Atari 2600 Phoenix Ape-X Score 224491.1 # 9
Atari Games Atari 2600 Pitfall! Ape-X Score -0.6 # 20
Atari Games Atari 2600 Pong Ape-X Score 20.9 # 17
Atari Games Atari 2600 Private Eye Ape-X Score 49.8 # 51
Atari Games Atari 2600 Q*Bert Ape-X Score 302391.3 # 5
Atari Games Atari 2600 River Raid Ape-X Score 63864.4 # 3
Atari Games Atari 2600 Road Runner Ape-X Score 222234.5 # 9
Atari Games Atari 2600 Robotank Ape-X Score 73.8 # 10
Atari Games Atari 2600 Seaquest Ape-X Score 392952.3 # 9
Atari Games Atari 2600 Skiing Ape-X Score -10789.9 # 3
Atari Games Atari 2600 Solaris Ape-X Score 2892.9 # 18
Atari Games Atari 2600 Space Invaders Ape-X Score 54681 # 6
Atari Games Atari 2600 Star Gunner Ape-X Score 434342.5 # 7
Atari Games Atari 2600 Surround Ape-X Score 7.1 # 9
Atari Games Atari 2600 Tennis Ape-X Score 23.9 # 4
Atari Games Atari 2600 Time Pilot Ape-X Score 87085 # 8
Atari Games Atari 2600 Tutankham Ape-X Score 272.6 # 13
Atari Games Atari 2600 Up and Down Ape-X Score 401884.3 # 9
Atari Games Atari 2600 Venture Ape-X Score 1813 # 10
Atari Games Atari 2600 Video Pinball Ape-X Score 565163.2 # 14
Atari Games Atari 2600 Wizard of Wor Ape-X Score 46204 # 8
Atari Games Atari 2600 Yars Revenge Ape-X Score 148594.8 # 9
Atari Games Atari 2600 Zaxxon Ape-X Score 42285.5 # 8

Methods