The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning

In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al., 2016) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than A3C (Mnih et al., 2016). Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting... The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones. Next, we introduce the \b{eta}-leave-one-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training. read more

PDF Abstract ICLR 2018 PDF ICLR 2018 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien Reactor 500M Score 12689.1 # 10
Atari Games Atari 2600 Amidar Reactor 500M Score 1015.8 # 23
Atari Games Atari 2600 Assault Reactor 500M Score 8323.3 # 18
Atari Games Atari 2600 Asterix Reactor 500M Score 205914.0 # 17
Atari Games Atari 2600 Asteroids Reactor 500M Score 3726.1 # 17
Atari Games Atari 2600 Atlantis Reactor 500M Score 302831.0 # 32
Atari Games Atari 2600 Bank Heist Reactor 500M Score 1259.7 # 12
Atari Games Atari 2600 Battle Zone Reactor 500M Score 64070.0 # 10
Atari Games Atari 2600 Beam Rider Reactor 500M Score 11033.4 # 28
Atari Games Atari 2600 Berzerk Reactor 500M Score 2303.1 # 12
Atari Games Atari 2600 Bowling Reactor 500M Score 81.0 # 14
Atari Games Atari 2600 Boxing Reactor 500M Score 99.4 # 13
Atari Games Atari 2600 Breakout Reactor 500M Score 514.8 # 17
Atari Games Atari 2600 Centipede Reactor 500M Score 3422.0 # 38
Atari Games Atari 2600 Chopper Command Reactor 500M Score 107779.0 # 8
Atari Games Atari 2600 Crazy Climber Reactor 500M Score 236422.0 # 6
Atari Games Atari 2600 Defender Reactor 500M Score 223025.0 # 9
Atari Games Atari 2600 Demon Attack Reactor 500M Score 115154.0 # 14
Atari Games Atari 2600 Double Dunk Reactor 500M Score 23.0 # 9
Atari Games Atari 2600 Enduro Reactor 500M Score 2224.2 # 12
Atari Games Atari-57 Reactor Medium Human-Normalized Score 187.0% # 12

Methods