Search Results for author:
Found 92 papers, 56 papers with code
“Other-Play” for Zero-Shot Coordination
We consider the problem of zero-shot coordination - constructing AI agents that can coordinate with novel partners they have not seen before (e. g. humans).
Multi-agent Reinforcement Learning
Reinforcement Learning (RL)
Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps
In this paper, we take a different approach and instead address the effect of nonstationarity by adapting the widely used Adam optimiser.
The Complexity Dynamics of Grokking
We investigate the phenomenon of generalization through the lens of compression.
Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games
We show that with NZSC training, RL agents can be trained to coordinate well with novel partners even when the (exact) problem setting of the coordination is not common knowledge.
Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
While large models trained with self-supervised learning on offline datasets have shown remarkable capabilities in text and image domains, achieving the same generalisation for agents that act in sequential decision problems remains an open challenge.
General Reinforcement Learning
Reinforcement Learning (RL)
+1
CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants
We introduce a multi-turn benchmark for evaluating personalised alignment in LLM-based AI assistants, focusing on their ability to handle user-provided safety-critical contexts.
TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation
We then show that STICK (Self-TICK) can be used to improve generation quality across multiple benchmarks via self-refinement and Best-of-N selection.
Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources
Our method improves performance by 25. 51% for TQA on WikiSQL and 22. 57% for MHQA on HotPotQA compared to the fine-tuned baselines.
JaxLife: An Open-Ended Agentic Simulator
Towards this, we present JaxLife: an artificial life simulator in which embodied agents, parameterized by deep neural networks, must learn to survive in an expressive world containing programmable systems.
No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery
What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning.
BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts' attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers.
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
This approach signifies the beginning of a new era in scientific discovery in machine learning: bringing the transformative benefits of AI agents to the entire research process of AI itself, and taking us closer to a world where endless affordable creativity and innovation can be unleashed on the world's most challenging problems.
Mixture of Experts in a Mixture of RL settings
Mixtures of Experts (MoEs) have gained prominence in (self-)supervised learning due to their enhanced inference efficiency, adaptability to distributed training, and modularity.
Behaviour Distillation
Filling the gap, we formalize behaviour distillation, a setting that aims to discover and then condense the information required for training an expert policy into a synthetic dataset of state-action pairs, without access to expert data.
Discovering Preference Optimization Algorithms with and for Large Language Models
Specifically, we iteratively prompt an LLM to propose and implement new preference optimization loss functions based on previously-evaluated performance metrics.
Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history.
Risks and Opportunities of Open-Source Generative AI
Applications of Generative AI (Gen AI) are expected to revolutionize a number of different areas, ranging from science & medicine to education.
PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetition
A promising approach is to use the LLM itself as the safeguard.
Select to Perfect: Imitating desired behavior from large multi-agent data
Desired characteristics for an AI agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories.
Near to Mid-term Risks and Opportunities of Open-Source Generative AI
In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education.
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs).
Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection
In this paper, we study the problem of out-of-distribution (OOD) detection in RL, which focuses on identifying situations at test time that RL agents have not encountered in their training environments.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
+3
Policy-Guided Diffusion
Our approach provides an effective alternative to autoregressive offline world models, opening the door to the controllable generation of synthetic training data.
JaxUED: A simple and useable UED library in Jax
We present JaxUED, an open-source library providing minimal dependency implementations of modern Unsupervised Environment Design (UED) algorithms in Jax.
Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Either they are too slow for meaningful research to be performed without enormous computational resources, like Crafter, NetHack and Minecraft, or they are not complex enough to pose a significant challenge, like Minigrid and Procgen.
Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse.
Refining Minimax Regret for Unsupervised Environment Design
In this work, we introduce Bayesian level-perfect MMR (BLP), a refinement of the minimax regret objective that overcomes this limitation.
The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
We show that this truncation of rollouts results in a set of edge-of-reach states at which we are effectively ``bootstrapping from the void.''
Model-based Reinforcement Learning
reinforcement-learning
+1
Recurrent Reinforcement Learning with Memoroids
We leverage memoroids to propose a batching method that improves sample efficiency, increases the return, and simplifies the implementation of recurrent loss functions in reinforcement learning.
Symmetry-Breaking Augmentations for Ad Hoc Teamwork
In many collaborative settings, artificial intelligence (AI) agents must be able to adapt to new teammates that use unknown or previously unobserved strategies.
Mixtures of Experts Unlock Parameter Scaling for Deep RL
The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size.
Analysing the Sample Complexity of Opponent Shaping
Providing theoretical guarantees for M-FOS is hard because A) there is little literature on theoretical sample complexity bounds for meta-reinforcement learning B) M-FOS operates in continuous state and action spaces, so theoretical analysis is challenging.
Scaling Opponent Shaping to High Dimensional Games
In multi-agent settings with mixed incentives, methods developed for zero-sum games have been shown to lead to detrimental outcomes.
JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading
Financial exchanges across the world use limit order books (LOBs) to process orders and match trades.
Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network
Overall, our results invite the use and extension of the model in the direction of autoregressive large financial models for the generation of high-frequency financial data and we commit to open-sourcing our code to facilitate future research.
Unbiased Decisions Reduce Regret: Adversarial Domain Adaptation for the Bank Loan Problem
We introduce adversarial optimism (AdOpt) to directly address bias in the training set using adversarial domain adaptation.
Learning Multi-Agent Communication with Contrastive Learning
By examining the relationship between messages sent and received, we propose to learn to communicate using contrastive learning to maximize the mutual information between messages of a given trajectory.
A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
Training multiple agents to coordinate is an essential problem with applications in robotics, game theory, economics, and social sciences.
Arbitrary Order Meta-Learning with Simple Population-Based Evolution
Meta-learning, the notion of learning to learn, enables learning systems to quickly and flexibly solve new tasks.
Structured State Space Models for In-Context Reinforcement Learning
We propose a modification to a variant of S4 that enables us to initialise and reset the hidden state in parallel, allowing us to tackle reinforcement learning tasks.
MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning
Open-ended learning methods that automatically generate a curriculum of increasingly challenging tasks serve as a promising avenue toward generally capable reinforcement learning agents.
Adversarial Cheap Talk
More specifically, we show that an ACT Adversary is capable of harming performance by interfering with the learner's function approximation, or instead helping the Victim's performance by outputting useful features.
Perfectly Secure Steganography Using Minimum Entropy Coupling
Steganography is the practice of encoding secret information into innocuous content in such a manner that an adversarial third party would not realize that there is hidden meaning.
Equivariant Networks for Zero-Shot Coordination
Successful coordination in Dec-POMDPs requires agents to adopt robust strategies and interpretable styles of play for their partner.
Human-AI Coordination via Human-Regularized Search and Learning
First, we show that our method outperforms experts when playing with a group of diverse human players in ad-hoc teams.
Discovered Policy Optimisation
We refer to the immediate result as Learnt Policy Optimisation (LPO).
An Investigation of the Bias-Variance Tradeoff in Meta-Gradients
Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms.
Grounding Aleatoric Uncertainty for Unsupervised Environment Design
Problematically, in partially-observable or stochastic settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment in the intended deployment setting, while curriculum learning necessarily shifts the training distribution.
Generalized Beliefs for Cooperative AI
Self-play is a common paradigm for constructing solutions in Markov games that can yield optimal policies in collaborative settings.
Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world
We introduce Nocturne, a new 2D driving simulator for investigating multi-agent coordination under partial observability.
Model-Free Opponent Shaping
In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner's dilemma (IPD).
COLA: Consistent Learning with Opponent-Learning Awareness
Finally, in an empirical evaluation on a set of general-sum games, we find that COLA finds prosocial solutions and that it converges under a wider range of learning rates than HOLA and LOLA.
Evolving Curricula with Regret-Based Environment Design
Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex.
Learning Intuitive Policies Using Action Features
An unaddressed challenge in multi-agent coordination is to enable AI agents to exploit the semantic relationships between the features of actions and the features of observations.
Mirror Learning: A Unifying Framework of Policy Optimisation
In contrast, in this paper we introduce a novel theoretical framework, named Mirror Learning, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO.
Lyapunov Exponents for Diversity in Differentiable Games
We generalize this idea to non-conservative, multi-agent gradient systems by proposing a method - denoted Generalized Ridge Rider (GRR) - for finding arbitrary bifurcation points.
Neural Pseudo-Label Optimism for the Bank Loan Problem
The lender only observes whether a customer will repay a loan if the loan is issued to begin with, and thus modeled decisions affect what data is available to the lender for future decisions.
Replay-Guided Adversarial Environment Design
Furthermore, our theory suggests a highly counterintuitive improvement to PLR: by stopping the agent from updating its policy on uncurated levels (training on less data), we can improve the convergence to Nash equilibria.
Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers
After carefully redesigning the empirical setup, we find that when tuning learning rates properly, pretrained transformers do outperform or match training from scratch in all of our tasks, but only as long as the entire model is finetuned.
Communicating via Markov Decision Processes
We contribute a theoretically grounded approach to MCGs based on maximum entropy reinforcement learning and minimum entropy coupling that we call MEME.
Centralized Model and Exploration Policy for Multi-Agent RL
We also learn a centralized exploration policy within our model that learns to collect additional data in state-action regions with high model uncertainty.
Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings
Search is an important tool for computing effective policies in single- and multi-agent environments, and has been crucial for achieving superhuman performance in several benchmark fully and partially observable games.
A New Formalism, Method and Open Issues for Zero-Shot Coordination
We introduce an extension of the algorithm, other-play with tie-breaking, and prove that it is optimal in the LFC problem and an equilibrium in the LFC game.
Quasi-Equivalence Discovery for Zero-Shot Emergent Communication
In contrast, in this work, we present a novel problem setting and the Quasi-Equivalence Discovery (QED) algorithm that allows for zero-shot coordination (ZSC), i. e., discovering protocols that can generalize to independently trained agents.
Off-Belief Learning
Policies learned through self-play may adopt arbitrary conventions and implicitly rely on multi-step reasoning based on fragile assumptions about other agents' actions and thus fail when paired with humans or independently trained agents at test time.
Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
In the era of ever decreasing loss functions, SGD and its various offspring have become the go-to optimization tool in machine learning and are a key component of the success of deep neural networks (DNNs).
Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations
Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels.
The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets
For neural models to garner widespread public trust and ensure fairness, we must have human-intelligible explanations for their predictions.
Compositionality and Capacity in Emergent Languages
Our hypothesis is that there should be a specific range of model capacity and channel bandwidth that induces compositional structure in the resulting language and consequently encourages systematic generalization.
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
"Other-Play" for Zero-Shot Coordination
We consider the problem of zero-shot coordination - constructing AI agents that can coordinate with novel partners they have not seen before (e. g. humans).
On the interaction between supervision and self-play in emergent communication
A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training.
Improving Policies via Search in Cooperative Partially Observable Games
The first one, single-agent search, effectively converts the problem into a single agent setting by making all but one of the agents play according to the agreed-upon policy.
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives.
Seeded self-play for language learning
Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations, including populations speaking human language.
Capacity, Bandwidth, and Compositionality in Emergent Language Learning
In this paper, we investigate the learning biases that affect the efficacy and compositionality of emergent languages.
Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
We aim for this framework to provide a publicly available, off-the-shelf evaluation when the feature-selection perspective on explanations is needed.
Modeling Fake News in Social Networks with Deep Multi-Agent Reinforcement Learning
We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to or takes over a subset of agents.
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives.
A Survey of Reinforcement Learning Informed by Natural Language
To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand.
Differentiable Game Mechanics
The decomposition motivates Symplectic Gradient Adjustment (SGA), a new algorithm for finding stable fixed points in differentiable games.
On the Pitfalls of Measuring Emergent Communication
How do we know if communication is emerging in a multi-agent system?
The StarCraft Multi-Agent Challenge
In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
Stable Opponent Shaping in Differentiable Games
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL.
A Better Baseline for Second Order Gradient Estimation in Stochastic Computation Graphs
To improve the sample efficiency of DiCE, we propose a new baseline term for higher order gradient estimation.
DiCE: The Infinitely Differentiable Monte Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted.
Ranked #1 on
SMAC+
on Off_Near_parallel
Multi-agent Reinforcement Learning
reinforcement-learning
+5
The Mechanics of n-Player Differentiable Games
The first is related to potential games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian games, a new class of games that obey a conservation law, akin to conservation laws in classical mechanical systems.
DiCE: The Infinitely Differentiable Monte-Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
Counterfactual Multi-Agent Policy Gradients
COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies.
Ranked #1 on
SMAC+
on Off_Superhard_parallel
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems.
