Prioritized Experience Replay

18 Nov 2015  ·  Tom Schaul, John Quan, Ioannis Antonoglou, David Silver ·

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.

PDF Abstract

Code
Add Remove Mark official

Results from the Paper
Edit

Ranked #3 on Atari Games on Atari 2600 Kangaroo

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien Prior noop Score 4203.8 # 21
Compare
Atari Games Atari 2600 Alien Prior hs Score 1334.7 # 35
Compare
Atari Games Atari 2600 Amidar Prior hs Score 129.1 # 45
Compare
Atari Games Atari 2600 Amidar Prior noop Score 1838.9 # 12
Compare
Atari Games Atari 2600 Assault Prior hs Score 6548.9 # 24
Compare
Atari Games Atari 2600 Assault Prior noop Score 7672.1 # 22
Compare
Atari Games Atari 2600 Asterix Prior noop Score 31527 # 21
Compare
Atari Games Atari 2600 Asterix Prior hs Score 22484.5 # 24
Compare
Atari Games Atari 2600 Asteroids Prior noop Score 2654.3 # 23
Compare
Atari Games Atari 2600 Asteroids Prior hs Score 1745.1 # 29
Compare
Atari Games Atari 2600 Atlantis Prior hs Score 330647.0 # 32
Compare
Atari Games Atari 2600 Atlantis Prior noop Score 357324.0 # 30
Compare
Atari Games Atari 2600 Bank Heist Prior noop Score 1054.6 # 23
Compare
Atari Games Atari 2600 Bank Heist Prior hs Score 876.6 # 31
Compare
Atari Games Atari 2600 Battle Zone Prior noop Score 31530.0 # 23
Compare
Atari Games Atari 2600 Battle Zone Prior hs Score 25520.0 # 31
Compare
Atari Games Atari 2600 Beam Rider Prior noop Score 23384.2 # 18
Compare
Atari Games Atari 2600 Beam Rider Prior hs Score 31181.3 # 13
Compare
Atari Games Atari 2600 Berzerk Prior noop Score 1305.6 # 23
Compare
Atari Games Atari 2600 Berzerk Prior hs Score 865.9 # 30
Compare
Atari Games Atari 2600 Bowling Prior hs Score 52 # 29
Compare
Atari Games Atari 2600 Bowling Prior noop Score 47.9 # 33
Compare
Atari Games Atari 2600 Boxing Prior noop Score 95.6 # 24
Compare
Atari Games Atari 2600 Boxing Prior hs Score 72.3 # 35
Compare
Atari Games Atari 2600 Breakout Prior noop Score 373.9 # 29
Compare
Atari Games Atari 2600 Breakout Prior hs Score 343.0 # 39
Compare
Atari Games Atari 2600 Centipede Prior noop Score 4463.2 # 35
Compare
Atari Games Atari 2600 Centipede Prior hs Score 3489.1 # 41
Compare
Atari Games Atari 2600 Chopper Command Prior hs Score 4635.0 # 34
Compare
Atari Games Atari 2600 Chopper Command Prior noop Score 8600.0 # 21
Compare
Atari Games Atari 2600 Crazy Climber Prior noop Score 141161.0 # 19
Compare
Atari Games Atari 2600 Crazy Climber Prior hs Score 127512.0 # 27
Compare
Atari Games Atari 2600 Demon Attack Prior hs Score 61277.5 # 29
Compare
Atari Games Atari 2600 Demon Attack Prior noop Score 71846.4 # 24
Compare
Atari Games Atari 2600 Double Dunk Prior hs Score 16.0 # 15
Compare
Atari Games Atari 2600 Double Dunk Prior noop Score 18.5 # 13
Compare
Atari Games Atari 2600 Enduro Prior hs Score 1831.0 # 22
Compare
Atari Games Atari 2600 Enduro Prior noop Score 2093.0 # 17
Compare
Atari Games Atari 2600 Fishing Derby Prior hs Score 9.8 # 32
Compare
Atari Games Atari 2600 Fishing Derby Prior noop Score 39.5 # 18
Compare
Atari Games Atari 2600 Freeway Prior noop Score 33.7 # 14
Compare
Atari Games Atari 2600 Freeway Prior hs Score 28.9 # 37
Compare
Atari Games Atari 2600 Frostbite Prior noop Score 4380.1 # 18
Compare
Atari Games Atari 2600 Frostbite Prior hs Score 3510.0 # 23
Compare
Atari Games Atari 2600 Gopher Prior noop Score 32487.2 # 20
Compare
Atari Games Atari 2600 Gopher Prior hs Score 34858.8 # 18
Compare
Atari Games Atari 2600 Gravitar Prior noop Score 548.5 # 29
Compare
Atari Games Atari 2600 Gravitar Prior hs Score 269.5 # 47
Compare
Atari Games Atari 2600 HERO Prior hs Score 20889.9 # 27
Compare
Atari Games Atari 2600 HERO Prior noop Score 23037.7 # 22
Compare
Atari Games Atari 2600 Ice Hockey Prior hs Score -0.2 # 20
Compare
Atari Games Atari 2600 Ice Hockey Prior noop Score 1.3 # 16
Compare
Atari Games Atari 2600 James Bond Prior noop Score 5148.0 # 14
Compare
Atari Games Atari 2600 James Bond Prior hs Score 3961.0 # 16
Compare
Atari Games Atari 2600 Kangaroo Prior noop Score 16200.0 # 3
Compare
Atari Games Atari 2600 Kangaroo Prior hs Score 12185.0 # 20
Compare
Atari Games Atari 2600 Krull Prior hs Score 6872.8 # 34
Compare
Atari Games Atari 2600 Krull Prior noop Score 9728.0 # 21
Compare
Atari Games Atari 2600 Kung-Fu Master Prior hs Score 31676.0 # 31
Compare
Atari Games Atari 2600 Kung-Fu Master Prior noop Score 39581.0 # 22
Compare
Atari Games Atari 2600 Montezuma's Revenge Prior hs Score 51 # 33
Compare
Atari Games Atari 2600 Ms. Pacman Prior noop Score 6518.7 # 13
Compare
Atari Games Atari 2600 Ms. Pacman Prior hs Score 1865.9 # 34
Compare
Atari Games Atari 2600 Name This Game Prior noop Score 12270.5 # 21
Compare
Atari Games Atari 2600 Name This Game Prior hs Score 10497.6 # 29
Compare
Atari Games Atari 2600 Pong Prior noop Score 20.6 # 24
Compare
Atari Games Atari 2600 Pong Prior hs Score 18.9 # 36
Compare
Atari Games Atari 2600 Private Eye Prior hs Score 670.7 # 24
Compare
Atari Games Atari 2600 Private Eye Prior noop Score 200.0 # 36
Compare
Atari Games Atari 2600 Q*Bert Prior noop Score 16256.5 # 23
Compare
Atari Games Atari 2600 Q*Bert Prior hs Score 9944 # 40
Compare
Atari Games Atari 2600 River Raid Prior hs Score 11807.2 # 26
Compare
Atari Games Atari 2600 River Raid Prior noop Score 14522.3 # 21
Compare
Atari Games Atari 2600 Road Runner Prior hs Score 52264.0 # 25
Compare
Atari Games Atari 2600 Road Runner Prior noop Score 57608.0 # 18
Compare
Atari Games Atari 2600 Robotank Prior noop Score 62.6 # 20
Compare
Atari Games Atari 2600 Robotank Prior hs Score 56.2 # 27
Compare
Atari Games Atari 2600 Seaquest Prior noop Score 26357.8 # 15
Compare
Atari Games Atari 2600 Seaquest Prior hs Score 25463.7 # 16
Compare
Atari Games Atari 2600 Space Invaders Prior hs Score 3912.1 # 22
Compare
Atari Games Atari 2600 Space Invaders Prior noop Score 2865.8 # 28
Compare
Atari Games Atari 2600 Star Gunner Prior noop Score 63302.0 # 24
Compare
Atari Games Atari 2600 Star Gunner Prior hs Score 61582.0 # 25
Compare
Atari Games Atari 2600 Tennis Prior noop Score 0.0 # 20
Compare
Atari Games Atari 2600 Tennis Prior hs Score -5.3 # 29
Compare
Atari Games Atari 2600 Time Pilot Prior hs Score 5963.0 # 33
Compare
Atari Games Atari 2600 Time Pilot Prior noop Score 9197.0 # 24
Compare
Atari Games Atari 2600 Tutankham Prior noop Score 204.6 # 24
Compare
Atari Games Atari 2600 Tutankham Prior hs Score 56.9 # 40
Compare
Atari Games Atari 2600 Up and Down Prior noop Score 16154.1 # 32
Compare
Atari Games Atari 2600 Up and Down Prior hs Score 12157.4 # 36
Compare
Atari Games Atari 2600 Venture Prior hs Score 94.0 # 33
Compare
Atari Games Atari 2600 Venture Prior noop Score 54.0 # 36
Compare
Atari Games Atari 2600 Video Pinball Prior hs Score 295972.8 # 24
Compare
Atari Games Atari 2600 Video Pinball Prior noop Score 282007.3 # 25
Compare
Atari Games Atari 2600 Wizard of Wor Prior hs Score 5727.0 # 30
Compare
Atari Games Atari 2600 Wizard of Wor Prior noop Score 4802.0 # 32
Compare
Atari Games Atari 2600 Zaxxon Prior noop Score 10469.0 # 25
Compare
Atari Games Atari 2600 Zaxxon Prior hs Score 9474.0 # 28
Compare

Methods
Add Remove

Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy