TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Unsupervised Reinforcement Learning	URLB (pixels, 10^5 frames)	SMM	Walker (mean normalized return)	6.07±6.14	# 10
Unsupervised Reinforcement Learning	URLB (pixels, 10^5 frames)	SMM	Quadruped (mean normalized return)	22.52±6.44	# 6
Unsupervised Reinforcement Learning	URLB (pixels, 10^5 frames)	SMM	Jaco (mean normalized return)	0.99±0.61	# 7
Unsupervised Reinforcement Learning	URLB (pixels, 10^6 frames)	SMM	Walker (mean normalized return)	6.61±6.70	# 9
Unsupervised Reinforcement Learning	URLB (pixels, 10^6 frames)	SMM	Quadruped (mean normalized return)	21.21±6.10	# 7
Unsupervised Reinforcement Learning	URLB (pixels, 10^6 frames)	SMM	Jaco (mean normalized return)	0.99±0.61	# 8
Unsupervised Reinforcement Learning	URLB (pixels, 2*10^6 frames)	SMM	Walker (mean normalized return)	6.61±6.70	# 10
Unsupervised Reinforcement Learning	URLB (pixels, 2*10^6 frames)	SMM	Quadruped (mean normalized return)	21.21±6.10	# 8
Unsupervised Reinforcement Learning	URLB (pixels, 2*10^6 frames)	SMM	Jaco (mean normalized return)	0.99±0.61	# 9
Unsupervised Reinforcement Learning	URLB (pixels, 5*10^5 frames)	SMM	Walker (mean normalized return)	6.31±6.44	# 9
Unsupervised Reinforcement Learning	URLB (pixels, 5*10^5 frames)	SMM	Quadruped (mean normalized return)	21.18±6.13	# 8
Unsupervised Reinforcement Learning	URLB (pixels, 5*10^5 frames)	SMM	Jaco (mean normalized return)	0.99±0.61	# 7
Unsupervised Reinforcement Learning	URLB (states, 10^5 frames)	SMM	Walker (mean normalized return)	57.84±26.88	# 9
Unsupervised Reinforcement Learning	URLB (states, 10^5 frames)	SMM	Quadruped (mean normalized return)	35.53±10.16	# 2
Unsupervised Reinforcement Learning	URLB (states, 10^5 frames)	SMM	Jaco (mean normalized return)	26.06±6.40	# 8
Unsupervised Reinforcement Learning	URLB (states, 10^6 frames)	SMM	Walker (mean normalized return)	72.60±32.07	# 8
Unsupervised Reinforcement Learning	URLB (states, 10^6 frames)	SMM	Quadruped (mean normalized return)	37.37±4.30	# 6
Unsupervised Reinforcement Learning	URLB (states, 10^6 frames)	SMM	Jaco (mean normalized return)	29.96±1.37	# 8
Unsupervised Reinforcement Learning	URLB (states, 2*10^6 frames)	SMM	Walker (mean normalized return)	77.13±29.55	# 2
Unsupervised Reinforcement Learning	URLB (states, 2*10^6 frames)	SMM	Quadruped (mean normalized return)	29.95±7.59	# 7
Unsupervised Reinforcement Learning	URLB (states, 2*10^6 frames)	SMM	Jaco (mean normalized return)	21.87±2.77	# 8
Unsupervised Reinforcement Learning	URLB (states, 5*10^5 frames)	SMM	Walker (mean normalized return)	73.64±33.56	# 8
Unsupervised Reinforcement Learning	URLB (states, 5*10^5 frames)	SMM	Quadruped (mean normalized return)	37.20±12.78	# 4
Unsupervised Reinforcement Learning	URLB (states, 5*10^5 frames)	SMM	Jaco (mean normalized return)	31.95±2.95	# 8

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-3)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-3?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-2)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-2?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-1)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-1?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-6)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-6?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-5)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-5?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-4)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-4?p=efficient-exploration-via-state-marginal)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/efficient-exploration-via-state-marginal/unsupervised-reinforcement-learning-on-urlb-7)](https://paperswithcode.com/sota/unsupervised-reinforcement-learning-on-urlb-7?p=efficient-exploration-via-state-marginal)`

Efficient Exploration via State Marginal Matching

12 Jun 2019 · Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, Ruslan Salakhutdinov ·

Exploration is critical to a reinforcement learning agent's performance in its given environment. Prior exploration methods are often based on using heuristic auxiliary predictions to guide policy behavior, lacking a mathematically-grounded objective with clear properties. In contrast, we recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution. The target distribution is a uniform distribution in most cases, but can incorporate prior knowledge if available. In effect, SMM amortizes the cost of learning to explore in a given environment. The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective. Using this formalism, we further demonstrate that prior work approximately maximizes the SMM objective, offering an explanation for the success of these methods. On both simulated and real-world tasks, we demonstrate that agents that directly optimize the SMM objective explore faster and adapt more quickly to new tasks as compared to prior exploration methods.

PDF Abstract

Code

Add Remove Mark official

RLAgent/state-marginal-matching

Tasks

Add Remove

Efficient Exploration

Unsupervised Reinforcement Learning

Datasets

URLB

Results from the Paper

Edit

Ranked #2 on Unsupervised Reinforcement Learning on URLB (states, 2*10^6 frames)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Unsupervised Reinforcement Learning	URLB (pixels, 10^5 frames)	SMM	Walker (mean normalized return)	6.07±6.14	# 10	Compare
			Quadruped (mean normalized return)	22.52±6.44	# 6	Compare
			Jaco (mean normalized return)	0.99±0.61	# 7	Compare
Unsupervised Reinforcement Learning	URLB (pixels, 10^6 frames)	SMM	Walker (mean normalized return)	6.61±6.70	# 9	Compare
			Quadruped (mean normalized return)	21.21±6.10	# 7	Compare
			Jaco (mean normalized return)	0.99±0.61	# 8	Compare
Unsupervised Reinforcement Learning	URLB (pixels, 2*10^6 frames)	SMM	Walker (mean normalized return)	6.61±6.70	# 10	Compare
			Quadruped (mean normalized return)	21.21±6.10	# 8	Compare
			Jaco (mean normalized return)	0.99±0.61	# 9	Compare
Unsupervised Reinforcement Learning	URLB (pixels, 5*10^5 frames)	SMM	Walker (mean normalized return)	6.31±6.44	# 9	Compare
			Quadruped (mean normalized return)	21.18±6.13	# 8	Compare
			Jaco (mean normalized return)	0.99±0.61	# 7	Compare
Unsupervised Reinforcement Learning	URLB (states, 10^5 frames)	SMM	Walker (mean normalized return)	57.84±26.88	# 9	Compare
			Quadruped (mean normalized return)	35.53±10.16	# 2	Compare
			Jaco (mean normalized return)	26.06±6.40	# 8	Compare
Unsupervised Reinforcement Learning	URLB (states, 10^6 frames)	SMM	Walker (mean normalized return)	72.60±32.07	# 8	Compare
			Quadruped (mean normalized return)	37.37±4.30	# 6	Compare
			Jaco (mean normalized return)	29.96±1.37	# 8	Compare
Unsupervised Reinforcement Learning	URLB (states, 2*10^6 frames)	SMM	Walker (mean normalized return)	77.13±29.55	# 2	Compare
			Quadruped (mean normalized return)	29.95±7.59	# 7	Compare
			Jaco (mean normalized return)	21.87±2.77	# 8	Compare
Unsupervised Reinforcement Learning	URLB (states, 5*10^5 frames)	SMM	Walker (mean normalized return)	73.64±33.56	# 8	Compare
			Quadruped (mean normalized return)	37.20±12.78	# 4	Compare
			Jaco (mean normalized return)	31.95±2.95	# 8	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

Efficient Exploration via State Marginal Matching

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove