Active World Model Learning in Agent-rich Environments with Progress Curiosity
World models are a family of predictive models that solve self-supervised problems on how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we simulate a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world stimuli. We propose an AWML system driven by $\gamma$-Progress: a scalable and effective learning progress-based curiosity signal. We show that $\gamma$-Progress is robust to "white noise" and naturally gives rise to an exploration policy that allocates attention in a balanced manner, with a preference towards agents displaying complex yet learnable behaviors. As a result, our $\gamma$-Progress driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.
PDF ICML 2020 PDF