Evolution Strategies as a Scalable Alternative to Reinforcement Learning

10 Mar 2017Tim SalimansJonathan HoXi ChenSzymon SidorIlya Sutskever

We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers... (read more)

PDF Abstract

Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Atari Games Atari 2600 Alien ES FF (1 hour) noop Score 994.0 # 14
Atari Games Atari 2600 Amidar ES FF (1 hour) noop Score 112.0 # 21
Atari Games Atari 2600 Assault ES FF (1 hour) noop Score 1673.9 # 19
Atari Games Atari 2600 Asterix ES FF (1 hour) noop Score 1440.0 # 20
Atari Games Atari 2600 Asteroids ES FF (1 hour) noop Score 1562.0 # 11
Atari Games Atari 2600 Atlantis ES FF (1 hour) noop Score 1267410.0 # 1
Atari Games Atari 2600 Bank Heist ES FF (1 hour) noop Score 225.0 # 20
Atari Games Atari 2600 Battle Zone ES FF (1 hour) noop Score 16600.0 # 17
Atari Games Atari 2600 Beam Rider ES FF (1 hour) noop Score 744.0 # 23
Atari Games Atari 2600 Berzerk ES FF (1 hour) noop Score 686.0 # 15
Atari Games Atari 2600 Bowling ES FF (1 hour) noop Score 30.0 # 21
Atari Games Atari 2600 Boxing ES FF (1 hour) noop Score 49.8 # 18
Atari Games Atari 2600 Breakout ES FF (1 hour) noop Score 9.5 # 21
Atari Games Atari 2600 Centipede ES FF (1 hour) noop Score 7783.9 # 6
Atari Games Atari 2600 Chopper Command ES FF (1 hour) noop Score 3710.0 # 17
Atari Games Atari 2600 Crazy Climber ES FF (1 hour) noop Score 26430.0 # 20
Atari Games Atari 2600 Demon Attack ES FF (1 hour) noop Score 1166.5 # 20
Atari Games Atari 2600 Double Dunk ES FF (1 hour) noop Score 0.2 # 6
Atari Games Atari 2600 Enduro ES FF (1 hour) noop Score 95.0 # 19
Atari Games Atari 2600 Fishing Derby ES FF (1 hour) noop Score -49.0 # 20
Atari Games Atari 2600 Freeway ES FF (1 hour) noop Score 31.0 # 8
Atari Games Atari 2600 Frostbite ES FF (1 hour) noop Score 370.0 # 20
Atari Games Atari 2600 Gopher ES FF (1 hour) noop Score 582.0 # 22
Atari Games Atari 2600 Gravitar ES FF (1 hour) noop Score 805.0 # 3
Atari Games Atari 2600 Ice Hockey ES FF (1 hour) noop Score -4.1 # 15
Atari Games Atari 2600 Kangaroo ES FF (1 hour) noop Score 11200.0 # 10
Atari Games Atari 2600 Krull ES FF (1 hour) noop Score 8647.2 # 7
Atari Games Atari 2600 Montezuma's Revenge ES FF (1 hour) noop Score 0.0 # 23
Atari Games Atari 2600 Name This Game ES FF (1 hour) noop Score 4503.0 # 20
Atari Games Atari 2600 Pong ES FF (1 hour) noop Score 21.0 # 1
Atari Games Atari 2600 Private Eye ES FF (1 hour) noop Score 100.0 # 21
Atari Games Atari 2600 Q*Bert ES FF (1 hour) noop Score 147.5 # 26
Atari Games Atari 2600 River Raid ES FF (1 hour) noop Score 5009.0 # 19
Atari Games Atari 2600 Road Runner ES FF (1 hour) noop Score 16590.0 # 20
Atari Games Atari 2600 Robotank ES FF (1 hour) noop Score 11.9 # 20
Atari Games Atari 2600 Seaquest ES FF (1 hour) noop Score 1390.0 # 19
Atari Games Atari 2600 Space Invaders ES FF (1 hour) noop Score 678.5 # 21
Atari Games Atari 2600 Star Gunner ES FF (1 hour) noop Score 1470.0 # 19
Atari Games Atari 2600 Tennis ES FF (1 hour) noop Score -4.5 # 12
Atari Games Atari 2600 Time Pilot ES FF (1 hour) noop Score 4970.0 # 16
Atari Games Atari 2600 Tutankham ES FF (1 hour) noop Score 130.3 # 12
Atari Games Atari 2600 Up and Down ES FF (1 hour) noop Score 67974.0 # 4
Atari Games Atari 2600 Venture ES FF (1 hour) noop Score 760.0 # 6
Atari Games Atari 2600 Video Pinball ES FF (1 hour) noop Score 22834.8 # 20
Atari Games Atari 2600 Wizard of Wor ES FF (1 hour) noop Score 3480.0 # 16
Atari Games Atari 2600 Zaxxon ES FF (1 hour) noop Score 6380.0 # 15