Recurrent Experience Replay in Distributed Reinforcement Learning

Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay. We study the effects of parameter lag resulting in representational drift and recurrent state staleness and empirically derive an improved training strategy. Using a single network architecture and fixed set of hyperparameters, the resulting agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and surpasses the state of the art on DMLab-30. It is the first agent to exceed human-level performance in 52 of the 57 Atari games.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Atari Games Atari 2600 Alien R2D2 Score 229496.9 # 4
Atari Games Atari 2600 Amidar R2D2 Score 29321.4 # 2
Atari Games Atari 2600 Assault R2D2 Score 108197.0 # 2
Atari Games Atari 2600 Asterix R2D2 Score 999153.3 # 2
Atari Games Atari 2600 Asteroids R2D2 Score 357867.7 # 5
Atari Games Atari 2600 Atlantis R2D2 Score 1620764.0 # 6
Atari Games Atari 2600 Bank Heist R2D2 Score 24235.9 # 2
Atari Games Atari 2600 Battle Zone R2D2 Score 751880.0 # 4
Atari Games Atari 2600 Beam Rider R2D2 Score 188257.4 # 5
Atari Games Atari 2600 Berzerk R2D2 Score 53318.7 # 5
Atari Games Atari 2600 Bowling R2D2 Score 219.5 # 4
Atari Games Atari 2600 Boxing R2D2 Score 98.5 # 20
Atari Games Atari 2600 Breakout R2D2 Score 837.7 # 7
Atari Games Atari 2600 Centipede R2D2 Score 599140.3 # 5
Atari Games Atari 2600 Chopper Command R2D2 Score 986652.0 # 6
Atari Games Atari 2600 Crazy Climber R2D2 Score 366690.7 # 3
Atari Games Atari 2600 Defender R2D2 Score 665792.0 # 7
Atari Games Atari 2600 Demon Attack R2D2 Score 140002.3 # 8
Atari Games Atari 2600 Double Dunk R2D2 Score 23.7 # 8
Atari Games Atari 2600 Enduro R2D2 Score 2372.7 # 6
Atari Games Atari 2600 Fishing Derby R2D2 Score 85.8 # 3
Atari Games Atari 2600 Freeway R2D2 Score 32.5 # 25
Atari Games Atari 2600 Frostbite R2D2 Score 315456.4 # 4
Atari Games Atari 2600 Gopher R2D2 Score 124776.3 # 4
Atari Games Atari 2600 Gravitar R2D2 Score 15680.7 # 2
Atari Games Atari 2600 HERO R2D2 Score 39537.1 # 3
Atari Games Atari 2600 Ice Hockey R2D2 Score 79.3 # 2
Atari Games Atari 2600 James Bond R2D2 Score 25354.0 # 10
Atari Games Atari 2600 Kangaroo R2D2 Score 14130.7 # 13
Atari Games Atari 2600 Krull R2D2 Score 218448.1 # 4
Atari Games Atari 2600 Kung-Fu Master R2D2 Score 233413.3 # 3
Atari Games Atari 2600 Montezuma's Revenge R2D2 Score 2061.3 # 18
Atari Games Atari 2600 Ms. Pacman R2D2 Score 42281.7 # 4
Atari Games Atari 2600 Name This Game R2D2 Score 58182.7 # 3
Atari Games Atari 2600 Phoenix R2D2 Score 864020.0 # 6
Atari Games Atari 2600 Pitfall! R2D2 Score 0.0 # 4
Atari Games Atari 2600 Pong R2D2 Score 21.0 # 1
Atari Games Atari 2600 Private Eye R2D2 Score 5322.7 # 13
Atari Games Atari 2600 Q*Bert R2D2 Score 408850.0 # 3
Atari Games Atari 2600 River Raid R2D2 Score 45632.1 # 5
Atari Games Atari 2600 Road Runner R2D2 Score 599246.7 # 5
Atari Games Atari 2600 Robotank R2D2 Score 100.4 # 7
Atari Games Atari 2600 Seaquest R2D2 Score 999996.7 # 4
Atari Games Atari 2600 Skiing R2D2 Score -30021.7 # 3
Atari Games Atari 2600 Solaris R2D2 Score 3787.2 # 15
Atari Games Atari 2600 Space Invaders R2D2 Score 43223.4 # 10
Atari Games Atari 2600 Star Gunner R2D2 Score 717344.0 # 2
Atari Games Atari 2600 Surround R2D2 Score 9.9 # 3
Atari Games Atari 2600 Tennis R2D2 Score -0.1 # 29
Atari Games Atari 2600 Time Pilot R2D2 Score 445377.3 # 3
Atari Games Atari 2600 Tutankham R2D2 Score 395.3 # 6
Atari Games Atari 2600 Up and Down R2D2 Score 589226.9 # 8
Atari Games Atari 2600 Venture R2D2 Score 1970.7 # 8
Atari Games Atari 2600 Video Pinball R2D2 Score 999383.2 # 1
Atari Games Atari 2600 Wizard of Wor R2D2 Score 144362.7 # 3
Atari Games Atari 2600 Yars Revenge R2D2 Score 995048.4 # 2
Atari Games Atari 2600 Zaxxon R2D2 Score 224910.7 # 3
Atari Games Atari-57 R2D2 Human World Record Breakthrough 15 # 6
Mean Human Normalized Score 3374.31% # 6
Atari Games atari game R2D2 Human World Record Breakthrough 15 # 6
Atari Games Atari games R2D2 Mean Human Normalized Score 3374.31% # 6

Methods