Self-Imitation Learning

ICML 2018  ·  Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee ·

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL significantly improves advantage actor-critic (A2C) on several hard exploration Atari games and is competitive to the state-of-the-art count-based exploration methods. We also show that SIL improves proximal policy optimization (PPO) on MuJoCo tasks.

PDF Abstract ICML 2018 PDF ICML 2018 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien A2C + SIL Score 2242.2 # 29
Atari Games Atari 2600 Amidar A2C + SIL Score 1362 # 20
Atari Games Atari 2600 Assault A2C + SIL Score 1812 # 37
Atari Games Atari 2600 Asterix A2C + SIL Score 17984.2 # 31
Atari Games Atari 2600 Asteroids A2C + SIL Score 2259.4 # 25
Atari Games Atari 2600 Atlantis A2C + SIL Score 3084781.7 # 3
Atari Games Atari 2600 Bank Heist A2C + SIL Score 1137.8 # 19
Atari Games Atari 2600 Battle Zone A2C + SIL Score 25075 # 32
Atari Games Atari 2600 Beam Rider A2C + SIL Score 2366.2 # 43
Atari Games Atari 2600 Bowling A2C + SIL Score 31.1 # 42
Atari Games Atari 2600 Boxing A2C + SIL Score 99.6 # 14
Atari Games Atari 2600 Breakout A2C + SIL Score 452 # 22
Atari Games Atari 2600 Centipede A2C + SIL Score 7559.5 # 26
Atari Games Atari 2600 Chopper Command A2C + SIL Score 6710 # 24
Atari Games Atari 2600 Crazy Climber A2C + SIL Score 130185.8 # 24
Atari Games Atari 2600 Demon Attack A2C + SIL Score 10140.5 # 39
Atari Games Atari 2600 Double Dunk A2C + SIL Score 21.5 # 12
Atari Games Atari 2600 Enduro A2C + SIL Score 1205.1 # 29
Atari Games Atari 2600 Fishing Derby A2C + SIL Score 55.8 # 11
Atari Games Atari 2600 Freeway A2C + SIL Score 32.2 # 27
Atari Games Atari 2600 Frostbite A2C + SIL Score 6289.8 # 14
Atari Games Atari 2600 Gopher A2C + SIL Score 23304.2 # 21
Atari Games Atari 2600 Gravitar A2C + SIL Score 1874.2 # 18
Atari Games Atari 2600 HERO A2C + SIL Score 33156.7 # 10
Atari Games Atari 2600 Ice Hockey A2C + SIL Score -2.4 # 20
Atari Games Atari 2600 James Bond A2C + SIL Score 310.8 # 42
Atari Games Atari 2600 Kangaroo A2C + SIL Score 2888.3 # 32
Atari Games Atari 2600 Krull A2C + SIL Score 10614.6 # 16
Atari Games Atari 2600 Kung-Fu Master A2C + SIL Score 34449.2 # 26
Atari Games Atari 2600 Montezuma's Revenge A2C + SIL Score 1100 # 21
Atari Games Atari 2600 Ms. Pacman A2C + SIL Score 4025.1 # 23
Atari Games Atari 2600 Name This Game A2C + SIL Score 14958.2 # 17
Atari Games Atari 2600 Pong A2C + SIL Score 20.9 # 17
Atari Games Atari 2600 Private Eye A2C + SIL Score 661.2 # 25
Atari Games Atari 2600 Q*Bert A2C + SIL Score 104975.6 # 6
Atari Games Atari 2600 River Raid A2C + SIL Score 14306.1 # 22
Atari Games Atari 2600 Road Runner A2C + SIL Score 57071.7 # 20
Atari Games Atari 2600 Robotank A2C + SIL Score 10.5 # 39
Atari Games Atari 2600 Seaquest A2C + SIL Score 2456.5 # 36
Atari Games Atari 2600 Space Invaders A2C + SIL Score 2951.7 # 26
Atari Games Atari 2600 Star Gunner A2C + SIL Score 31309.2 # 35
Atari Games Atari 2600 Tennis A2C + SIL Score -17.3 # 29
Atari Games Atari 2600 Time Pilot A2C + SIL Score 10811.7 # 22
Atari Games Atari 2600 Tutankham A2C + SIL Score 340.5 # 8
Atari Games Atari 2600 Up and Down A2C + SIL Score 53314.6 # 22
Atari Games Atari 2600 Venture A2C + SIL Score 0 # 49
Atari Games Atari 2600 Video Pinball A2C + SIL Score 461522.4 # 19
Atari Games Atari 2600 Wizard of Wor A2C + SIL Score 7088.3 # 26
Atari Games Atari 2600 Zaxxon A2C + SIL Score 9164.2 # 30

Methods


No methods listed for this paper. Add relevant methods here