QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

PDF Abstract ICML 2018 PDF ICML 2018 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
SMAC+ Def_Armored_parallel QMIX Median Win Rate 75.0 # 2
SMAC+ Def_Armored_sequential QMIX Median Win Rate 0.0 # 9
SMAC+ Def_Infantry_parallel QMIX Median Win Rate 95.0 # 3
SMAC+ Def_Infantry_sequential QMIX Median Win Rate 96.9 # 5
SMAC+ Def_Outnumbered_parallel QMIX Median Win Rate 30.0 # 2
SMAC+ Def_Outnumbered_sequential QMIX Median Win Rate 0.0 # 5
SMAC+ Off_Complicated_parallel QMIX Median Win Rate 0.0 # 4
SMAC+ Off_Complicated_sequential QMIX Median Win Rate 87.5 # 2
SMAC+ Off_Distant_parallel QMIX Median Win Rate 0.0 # 3
SMAC+ Off_Distant_sequential QMIX Median Win Rate 93.8 # 2
SMAC+ Off_Hard_parallel QMIX Median Win Rate 0.0 # 3
SMAC+ Off_Hard_sequential QMIX Median Win Rate 96.9 # 1
SMAC+ Off_Near_parallel QMIX Median Win Rate 95.0 # 1
SMAC+ Off_Near_sequential QMIX Median Win Rate 90.6 # 2
SMAC+ Off_Superhard_parallel QMIX Median Win Rate 0.0 # 1
SMAC+ Off_Superhard_sequential QMIX Median Win Rate 0.0 # 2
Starcraft II SMAC-Exp QMIX Median Win Rate % # 1


No methods listed for this paper. Add relevant methods here