Search Results for author:
Found 72 papers, 41 papers with code
“Other-Play” for Zero-Shot Coordination
We consider the problem of zero-shot coordination - constructing AI agents that can coordinate with novel partners they have not seen before (e. g. humans).
Multi-agent Reinforcement Learning
Reinforcement Learning (RL)
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs).
Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection
In this paper, we study the problem of out-of-distribution (OOD) detection in RL, which focuses on identifying situations at test time that RL agents have not encountered in their training environments.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
+2
Policy-Guided Diffusion
Our approach provides an effective alternative to autoregressive offline world models, opening the door to the controllable generation of synthetic training data.
JaxUED: A simple and useable UED library in Jax
We present JaxUED, an open-source library providing minimal dependency implementations of modern Unsupervised Environment Design (UED) algorithms in Jax.
Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Either they are too slow for meaningful research to be performed without enormous computational resources, like Crafter, NetHack and Minecraft, or they are not complex enough to pose a significant challenge, like Minigrid and Procgen.
Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to user inputs is of paramount importance.
Refining Minimax Regret for Unsupervised Environment Design
In this work, we introduce Bayesian level-perfect MMR (BLP), a refinement of the minimax regret objective that overcomes this limitation.
Revisiting Recurrent Reinforcement Learning with Memory Monoids
Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states.
Symmetry-Breaking Augmentations for Ad Hoc Teamwork
In many collaborative settings, artificial intelligence (AI) agents must be able to adapt to new teammates that use unknown or previously unobserved strategies.
Mixtures of Experts Unlock Parameter Scaling for Deep RL
The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size.
Analysing the Sample Complexity of Opponent Shaping
Providing theoretical guarantees for M-FOS is hard because A) there is little literature on theoretical sample complexity bounds for meta-reinforcement learning B) M-FOS operates in continuous state and action spaces, so theoretical analysis is challenging.
Scaling Opponent Shaping to High Dimensional Games
In multi-agent settings with mixed incentives, methods developed for zero-sum games have been shown to lead to detrimental outcomes.
JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading
Financial exchanges across the world use limit order books (LOBs) to process orders and match trades.
Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network
Overall, our results invite the use and extension of the model in the direction of autoregressive large financial models for the generation of high-frequency financial data and we commit to open-sourcing our code to facilitate future research.
Unbiased Decisions Reduce Regret: Adversarial Domain Adaptation for the Bank Loan Problem
We introduce adversarial optimism (AdOpt) to directly address bias in the training set using adversarial domain adaptation.
Learning Multi-Agent Communication with Contrastive Learning
By examining the relationship between messages sent and received, we propose to learn to communicate using contrastive learning to maximize the mutual information between messages of a given trajectory.
A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
Training multiple agents to coordinate is an essential problem with applications in robotics, game theory, economics, and social sciences.
Arbitrary Order Meta-Learning with Simple Population-Based Evolution
Meta-learning, the notion of learning to learn, enables learning systems to quickly and flexibly solve new tasks.
Structured State Space Models for In-Context Reinforcement Learning
We propose a modification to a variant of S4 that enables us to initialise and reset the hidden state in parallel, allowing us to tackle reinforcement learning tasks.
MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning
Open-ended learning methods that automatically generate a curriculum of increasingly challenging tasks serve as a promising avenue toward generally capable reinforcement learning agents.
Adversarial Cheap Talk
More specifically, we show that an ACT Adversary is capable of harming performance by interfering with the learner's function approximation, or instead helping the Victim's performance by outputting useful features.
Perfectly Secure Steganography Using Minimum Entropy Coupling
Steganography is the practice of encoding secret information into innocuous content in such a manner that an adversarial third party would not realize that there is hidden meaning.
Equivariant Networks for Zero-Shot Coordination
Successful coordination in Dec-POMDPs requires agents to adopt robust strategies and interpretable styles of play for their partner.
Discovered Policy Optimisation
We refer to the immediate result as Learnt Policy Optimisation (LPO).
Human-AI Coordination via Human-Regularized Search and Learning
First, we show that our method outperforms experts when playing with a group of diverse human players in ad-hoc teams.
An Investigation of the Bias-Variance Tradeoff in Meta-Gradients
Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms.
Grounding Aleatoric Uncertainty for Unsupervised Environment Design
Problematically, in partially-observable or stochastic settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment in the intended deployment setting, while curriculum learning necessarily shifts the training distribution.
Generalized Beliefs for Cooperative AI
Self-play is a common paradigm for constructing solutions in Markov games that can yield optimal policies in collaborative settings.
Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world
We introduce Nocturne, a new 2D driving simulator for investigating multi-agent coordination under partial observability.
Model-Free Opponent Shaping
In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner's dilemma (IPD).
COLA: Consistent Learning with Opponent-Learning Awareness
Finally, in an empirical evaluation on a set of general-sum games, we find that COLA finds prosocial solutions and that it converges under a wider range of learning rates than HOLA and LOLA.
Evolving Curricula with Regret-Based Environment Design
Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex.
Learning Intuitive Policies Using Action Features
An unaddressed challenge in multi-agent coordination is to enable AI agents to exploit the semantic relationships between the features of actions and the features of observations.
Mirror Learning: A Unifying Framework of Policy Optimisation
In contrast, in this paper we introduce a novel theoretical framework, named Mirror Learning, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO.
Lyapunov Exponents for Diversity in Differentiable Games
We generalize this idea to non-conservative, multi-agent gradient systems by proposing a method - denoted Generalized Ridge Rider (GRR) - for finding arbitrary bifurcation points.
Neural Pseudo-Label Optimism for the Bank Loan Problem
The lender only observes whether a customer will repay a loan if the loan is issued to begin with, and thus modeled decisions affect what data is available to the lender for future decisions.
Replay-Guided Adversarial Environment Design
Furthermore, our theory suggests a highly counterintuitive improvement to PLR: by stopping the agent from updating its policy on uncurated levels (training on less data), we can improve the convergence to Nash equilibria.
Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers
After carefully redesigning the empirical setup, we find that when tuning learning rates properly, pretrained transformers do outperform or match training from scratch in all of our tasks, but only as long as the entire model is finetuned.
Communicating via Markov Decision Processes
We contribute a theoretically grounded approach to MCGs based on maximum entropy reinforcement learning and minimum entropy coupling that we call MEME.
Centralized Model and Exploration Policy for Multi-Agent RL
We also learn a centralized exploration policy within our model that learns to collect additional data in state-action regions with high model uncertainty.
Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings
Search is an important tool for computing effective policies in single- and multi-agent environments, and has been crucial for achieving superhuman performance in several benchmark fully and partially observable games.
A New Formalism, Method and Open Issues for Zero-Shot Coordination
We introduce an extension of the algorithm, other-play with tie-breaking, and prove that it is optimal in the LFC problem and an equilibrium in the LFC game.
Quasi-Equivalence Discovery for Zero-Shot Emergent Communication
In contrast, in this work, we present a novel problem setting and the Quasi-Equivalence Discovery (QED) algorithm that allows for zero-shot coordination (ZSC), i. e., discovering protocols that can generalize to independently trained agents.
Off-Belief Learning
Policies learned through self-play may adopt arbitrary conventions and implicitly rely on multi-step reasoning based on fragile assumptions about other agents' actions and thus fail when paired with humans or independently trained agents at test time.
Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
In the era of ever decreasing loss functions, SGD and its various offspring have become the go-to optimization tool in machine learning and are a key component of the success of deep neural networks (DNNs).
Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations
Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels.
The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets
For neural models to garner widespread public trust and ensure fairness, we must have human-intelligible explanations for their predictions.
Compositionality and Capacity in Emergent Languages
Our hypothesis is that there should be a specific range of model capacity and channel bandwidth that induces compositional structure in the resulting language and consequently encourages systematic generalization.
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
"Other-Play" for Zero-Shot Coordination
We consider the problem of zero-shot coordination - constructing AI agents that can coordinate with novel partners they have not seen before (e. g. humans).
On the interaction between supervision and self-play in emergent communication
A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training.
Improving Policies via Search in Cooperative Partially Observable Games
The first one, single-agent search, effectively converts the problem into a single agent setting by making all but one of the agents play according to the agreed-upon policy.
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives.
Seeded self-play for language learning
Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations, including populations speaking human language.
Capacity, Bandwidth, and Compositionality in Emergent Language Learning
In this paper, we investigate the learning biases that affect the efficacy and compositionality of emergent languages.
Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
We aim for this framework to provide a publicly available, off-the-shelf evaluation when the feature-selection perspective on explanations is needed.
Modeling Fake News in Social Networks with Deep Multi-Agent Reinforcement Learning
We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to or takes over a subset of agents.
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives.
A Survey of Reinforcement Learning Informed by Natural Language
To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand.
Differentiable Game Mechanics
The decomposition motivates Symplectic Gradient Adjustment (SGA), a new algorithm for finding stable fixed points in differentiable games.
On the Pitfalls of Measuring Emergent Communication
How do we know if communication is emerging in a multi-agent system?
The StarCraft Multi-Agent Challenge
In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
Stable Opponent Shaping in Differentiable Games
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL.
A Better Baseline for Second Order Gradient Estimation in Stochastic Computation Graphs
To improve the sample efficiency of DiCE, we propose a new baseline term for higher order gradient estimation.
DiCE: The Infinitely Differentiable Monte Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted.
Ranked #1 on
SMAC+
on Off_Near_parallel
Multi-agent Reinforcement Learning
reinforcement-learning
+4
The Mechanics of n-Player Differentiable Games
The first is related to potential games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian games, a new class of games that obey a conservation law, akin to conservation laws in classical mechanical systems.
DiCE: The Infinitely Differentiable Monte-Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
Counterfactual Multi-Agent Policy Gradients
COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies.
Ranked #1 on
SMAC+
on Off_Superhard_parallel
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems.
