Search Results for author:
Found 121 papers, 66 papers with code
Policy-Guided Diffusion
Our approach provides an effective alternative to autoregressive offline world models, opening the door to the controllable generation of synthetic training data.
SplAgger: Split Aggregation for Meta-Reinforcement Learning
However, it remains unclear whether task inference sequence models are beneficial even when task inference objectives are not.
Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control
Learning a universal policy across different robot morphologies can significantly improve learning efficiency and enable zero-shot generalization to unseen morphologies.
Discovering Temporally-Aware Reinforcement Learning Algorithms
We propose a simple augmentation to two existing objective discovery approaches that allows the discovered algorithm to dynamically update its objective function throughout the agent's training procedure, resulting in expressive schedules and increased generalization across different training horizons.
JaxMARL: Multi-Agent RL Environments in JAX
This not only enables GPU acceleration, but also provides a more flexible MARL environment, unlocking the potential for self-play, meta-learning, and other future applications in MARL.
Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design
Recently, it has been shown that it is possible to meta-learn update rules, with the hope of discovering algorithms that can perform well on a wide range of RL tasks.
Recurrent Hypernetworks are Surprisingly Strong in Meta-RL
While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline.
Hierarchical Imitation Learning for Stochastic Environments
However, such methods are often inappropriate for stochastic environments where the agent must also react to external factors: because agent types are inferred from the observed future trajectory during training, these environments require that the contributions of internal and external factors to the agent behaviour are disentangled and only internal factors, i. e., those under the agent's control, are encoded in the type.
Bayesian Exploration Networks
Empirical results demonstrate that BEN can learn true Bayes-optimal policies in tasks where existing model-free approaches fail.
The Waymo Open Sim Agents Challenge
Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development.
Cheap Talk Discovery and Utilization in Multi-Agent Reinforcement Learning
By enabling agents to communicate, recent cooperative multi-agent reinforcement learning (MARL) methods have demonstrated better task performance and more coordinated behavior.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Why Target Networks Stabilise Temporal Difference Methods
Integral to recent successes in deep reinforcement learning has been a class of temporal difference methods that use infrequently updated target values for policy evaluation in a Markov Decision Process.
Universal Morphology Control via Contextual Modulation
Learning a universal policy across different robot morphologies can significantly improve learning efficiency and generalization in continuous control.
Trust-Region-Free Policy Optimization for Stochastic Policies
In this paper, we show that the trust region constraint over policies can be safely substituted by a trust-region-free constraint without compromising the underlying monotonic improvement guarantee.
A Survey of Meta-Reinforcement Learning
Meta-RL is most commonly studied in a problem setting where, given a distribution of tasks, the goal is to learn a policy that is capable of adapting to any new task from the task distribution with as little data as possible.
Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios
To our knowledge, this is the first application of a combined imitation and reinforcement learning approach in autonomous driving that utilizes large amounts of real-world human driving data.
SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning
In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies.
Particle-Based Score Estimation for State Space Model Learning in Autonomous Driving
Recent methods addressing this problem typically differentiate through time in a particle filter, which requires workarounds to the non-differentiable resampling step, that yield biased or high variance gradient estimates.
Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula
However, this approach produces agents that do not perform robustly in safety-critical settings, an issue that cannot be addressed by simply adding more data to the training set - we show that an agent trained using only a 10% subset of the data performs just as well as an agent trained on the entire dataset.
Equivariant Networks for Zero-Shot Coordination
Successful coordination in Dec-POMDPs requires agents to adopt robust strategies and interpretable styles of play for their partner.
Hypernetworks in Meta-Reinforcement Learning
In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, and that naive initializations yield poor performance; 2) propose a novel hypernetwork initialization scheme that matches or exceeds the performance of a state-of-the-art approach proposed for supervised settings, as well as being simpler and more general; and 3) use this method to show that hypernetworks can improve performance in meta-RL by evaluating on multiple simulated robotics benchmarks.
Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving
We demonstrate the first large-scale application of model-based generative adversarial imitation learning (MGAIL) to the task of dense urban self-driving.
An Investigation of the Bias-Variance Tradeoff in Meta-Gradients
Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms.
Generalized Beliefs for Cooperative AI
Self-play is a common paradigm for constructing solutions in Markov games that can yield optimal policies in collaborative settings.
Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation
The beam search refines these policies on the fly by pruning branches that are unfavourably evaluated by a discriminator.
Trust Region Bounds for Decentralized PPO Under Non-stationarity
We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary.
You May Not Need Ratio Clipping in PPO
Furthermore, we show that ESPO can be easily scaled up to distributed training with many workers, delivering strong performance as well.
Generalization in Cooperative Multi-Agent Systems
Specifically, we study generalization bounds under a linear dependence of the underlying dynamics on the agent capabilities, which can be seen as a generalization of Successor Features to MAS.
In Defense of the Unitary Scalarization for Deep Multi-Task Learning
We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings.
Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency
Sample efficiency is crucial for imitation learning methods to be applicable in real-world applications.
Regularized Softmax Deep Multi-Agent Q-Learning
Tackling overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting.
On the Practical Consistency of Meta-Reinforcement Learning Algorithms
We further find that theoretically inconsistent algorithms can be made consistent by continuing to update all agent components on the OOD tasks, and adapt as well or better than originally consistent ones.
Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
We present an extended abstract for the previously published work TESSERACT [Mahajan et al., 2021], which proposes a novel solution for Reinforcement Learning (RL) in large, factored action spaces using tensor decompositions.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Model based Multi-agent Reinforcement Learning with Tensor Decompositions
A challenge in multi-agent reinforcement learning is to be able to generalize over intractable state-action spaces.
Model-based Reinforcement Learning
Multi-agent Reinforcement Learning
+3
Stability and Generalisation in Batch Reinforcement Learning
Overfitting has been recently acknowledged as a key limiting factor in the capabilities of reinforcement learning algorithms, despite little theoretical characterisation.
Truncated Emphatic Temporal Difference Methods for Prediction and Control
Despite the theoretical success of emphatic TD methods in addressing the notorious deadly triad of off-policy RL, there are still two open problems.
Communicating via Markov Decision Processes
We contribute a theoretically grounded approach to MCGs based on maximum entropy reinforcement learning and minimum entropy coupling that we call MEME.
Bayesian Bellman Operators
We introduce a novel perspective on Bayesian reinforcement learning (RL); whereas existing approaches infer a posterior over the transition distribution or Q-function, we characterise the uncertainty in the Bellman operator.
SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching
We present SoftDICE, which achieves state-of-the-art performance for imitation learning.
Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning
Algorithms derived from Tesseract decompose the Q-tensor across agents and utilise low-rank tensor approximations to model agent interactions relevant to the task.
Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
Policy gradient methods are an attractive approach to multi-agent reinforcement learning problems due to their convergence properties and robustness in partially observable scenarios.
Regularized Softmax Deep Multi-Agent $Q$-Learning
Tackling overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting.
Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing
Recent research has shown that graph neural networks (GNNs) can learn policies for locomotion control that are as effective as a typical multi-layer perceptron (MLP), with superior transfer and multi-task performance (Wang et al., 2018; Huang et al., 2020).
Breaking the Deadly Triad with a Target Network
The deadly triad refers to the instability of a reinforcement learning algorithm when it employs off-policy learning, function approximation, and bootstrapping simultaneously.
Deep Interactive Bayesian Reinforcement Learning via Meta-Learning
The optimal adaptive behaviour under uncertainty over the other agents' strategies w. r. t.
Average-Reward Off-Policy Policy Evaluation with Function Approximation
We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function.
Can Q-Learning with Graph Networks Learn a Generalizable Branching Heuristic for a SAT Solver?
While more work is needed to apply Graph-Q-SAT to reduce wall clock time in modern SAT solving settings, it is a compelling proof-of-concept showing that RL equipped with Graph Neural Networks can learn a generalizable branching heuristic for SAT search.
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function.
UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
VDN and QMIX are two popular value-based algorithms for cooperative MARL that learn a centralized action value function as a monotonic mixing of per-agent utilities.
Multi-agent Reinforcement Learning
reinforcement-learning
+3
My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
They also allow practitioners to inject biases encoded in the structure of the input graph.
RODE: Learning Roles to Decompose Multi-Agent Tasks
Learning a role selector based on action effects makes role discovery much easier because it forms a bi-level learning hierarchy -- the role selector searches in a smaller role space and at a lower temporal resolution, while role policies learn in significantly reduced primitive action-observation spaces.
A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
In the second scenario, we consider optimizing a discounted objective ($\gamma < 1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.
Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
To rapidly learn a new task, it is often essential for agents to explore efficiently -- especially when performance matters from the first timestep.
Exploiting Submodular Value Functions For Scaling Up Active Perception
Furthermore, we show that, under certain conditions, including submodularity, the value function computed using greedy PBVI is guaranteed to have bounded error with respect to the optimal value function.
Real-Time Resource Allocation for Tracking Systems
Automated tracking is key to many computer vision applications.
WordCraft: An Environment for Benchmarking Commonsense Agents
This is partly due to the lack of lightweight simulation environments that sufficiently reflect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment.
Learning Retrospective Knowledge with Reverse Reinforcement Learning
We present a Reverse Reinforcement Learning (Reverse RL) approach for representing retrospective knowledge.
Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
We show in particular that this projection can fail to recover the optimal policy even with access to $Q^*$, which primarily stems from the equal weighting placed on each joint action.
Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments.
Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities; however, common patterns of behavior often emerge among these agents/entities.
Privileged Information Dropout in Reinforcement Learning
Using privileged information during training can improve the sample efficiency and performance of machine learning systems.
Maximizing Information Gain in Partially Observable Environments via Prediction Reward
Information gathering in a partially observable environment can be formulated as a reinforcement learning (RL), problem where the reward depends on the agent's uncertainty.
Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable.
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
FACMAC: Factored Multi-Agent Centralised Policy Gradients
We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces.
Optimistic Exploration even with a Pessimistic Initialisation
We show that this scheme is provably efficient in the tabular setting and extend it to the deep RL setting.
Reinforcement Learning Enhanced Quantum-inspired Algorithm for Combinatorial Optimization
Quantum hardware and quantum-inspired algorithms are becoming increasingly popular for combinatorial optimization.
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
Namely, the optimization problem in GenDICE is not a convex-concave saddle-point problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so any primal-dual algorithm is not guaranteed to converge or find the desired solution.
Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Framework
Interactive reinforcement learning provides a way for agents to learn to solve tasks from evaluative feedback provided by a human user.
Fast Efficient Hyperparameter Tuning for Policy Gradient Methods
The main idea is to use existing trajectories sampled by the policy gradient method to optimise a one-step improvement objective, yielding a sample and computationally efficient algorithm that is easy to implement.
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives.
VIABLE: Fast Adaptation via Backpropagating Learned Loss
In few-shot learning, typically, the loss function which is applied at test time is the one we are ultimately interested in minimising, such as the mean-squared-error loss for a regression problem.
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
With the help of the emphasis critic and the canonical value function critic, we show convergence for COF-PAC, where the critics are linear and the actor can be nonlinear.
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning.
MAVEN: Multi-Agent Variational Exploration
We specifically focus on QMIX [40], the current state-of-the-art in this domain.
Deep Coordination Graphs
This paper introduces the deep coordination graph (DCG) for collaborative multi-agent reinforcement learning.
Can $Q$-Learning with Graph Networks Learn a Generalizable Branching Heuristic for a SAT Solver?
While more work is needed to apply Graph-$Q$-SAT to reduce wall clock time in modern SAT solving settings, it is a compelling proof-of-concept showing that RL equipped with Graph Neural Networks can learn a generalizable branching heuristic for SAT search.
Improving SAT Solver Heuristics with Graph Networks and Reinforcement Learning
We present GQSAT, a branching heuristic in a Boolean SAT solver trained with value-based reinforcement learning (RL) using Graph Neural Networks for function approximation.
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives.
Growing Action Spaces
In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress.
A Survey of Reinforcement Learning Informed by Natural Language
To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand.
Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
This paper investigates the use of intrinsic reward to guide exploration in multi-agent reinforcement learning.
Deep Residual Reinforcement Learning
We revisit residual algorithms in both model-free and model-based reinforcement learning settings.
Model-based Reinforcement Learning
reinforcement-learning
+1
DAC: The Double Actor-Critic Architecture for Learning Options
We reformulate the option framework as two parallel augmented MDPs.
Multitask Soft Option Learning
We present Multitask Soft Option Learning(MSOL), a hierarchical multitask framework based on Planning as Inference.
Generalized Off-Policy Actor-Critic
We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting.
Fast Efficient Hyperparameter Tuning for Policy Gradients
The main idea is to use existing trajectories sampled by the policy gradient method to optimise a one-step improvement objective, yielding a sample and computationally efficient algorithm that is easy to implement.
The StarCraft Multi-Agent Challenge
In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap.
Ranked #6 on
SMAC
on SMAC 6h_vs_8z
Stable Opponent Shaping in Differentiable Games
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL.
Learning from Demonstration in the Wild
Learning from demonstration (LfD) is useful in settings where hand-coding behaviour or a reward function is impractical.
Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning
We present the Bayesian action decoder (BAD), a new multi-agent learning method that uses an approximate Bayesian update to obtain a public belief that conditions on the actions taken by all agents in the environment.
Multi-agent Reinforcement Learning
Policy Gradient Methods
+2
VIREL: A Variational Inference Framework for Reinforcement Learning
This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps.
Multi-Agent Common Knowledge Reinforcement Learning
In this paper, we show that common knowledge between agents allows for complex decentralised coordination.
Multi-agent Reinforcement Learning
reinforcement-learning
+3
Fast Context Adaptation via Meta-Learning
We propose CAVIA for meta-learning, a simple extension to MAML that is less prone to meta-overfitting, easier to parallelise, and more interpretable.
CAML: Fast Context Adaptation via Meta-Learning
We propose CAML, a meta-learning method for fast adaptation that partitions the model parameters into two parts: context parameters that serve as additional input to the model and are adapted on individual tasks, and shared parameters that are meta-trained and shared across tasks.
A Better Baseline for Second Order Gradient Estimation in Stochastic Computation Graphs
To improve the sample efficiency of DiCE, we propose a new baseline term for higher order gradient estimation.
DiCE: The Infinitely Differentiable Monte Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
Deep Variational Reinforcement Learning for POMDPs
Many real-world sequential decision making problems are partially observable by nature, and the environment model is typically unknown.
Fingerprint Policy Optimisation for Robust Reinforcement Learning
Policy gradient methods ignore the potential value of adjusting environment variables: unobservable state features that are randomly determined by the environment in a physical setting, but are controllable in a simulator.
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted.
Ranked #1 on
SMAC+
on Off_Near_parallel
Multi-agent Reinforcement Learning
reinforcement-learning
+4
TACO: Learning Task Decomposition via Temporal Alignment for Control
Many advanced Learning from Demonstration (LfD) methods consider the decomposition of complex, real-world tasks into simpler sub-tasks.
Fourier Policy Gradients
We propose a new way of deriving policy gradient updates for reinforcement learning.
DiCE: The Infinitely Differentiable Monte-Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
Expected Policy Gradients for Reinforcement Learning
For Gaussian policies, we introduce an exploration method that uses covariance proportional to the matrix exponential of the scaled Hessian of the critic with respect to the actions.
Dynamic-Depth Context Tree Weighting
Reinforcement learning (RL) in partially observable settings is challenging because the agent’s observations are not Markov.
TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
To address these challenges, we propose TreeQN, a differentiable, recursive, tree-structured model that serves as a drop-in replacement for any value function network in deep RL with discrete actions.
Learning with Opponent-Learning Awareness
We also show that the LOLA update rule can be efficiently calculated using an extension of the policy gradient estimator, making the method suitable for model-free RL.
Expected Policy Gradients
We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning.
Counterfactual Multi-Agent Policy Gradients
COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies.
Ranked #1 on
SMAC+
on Off_Superhard_parallel
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems.
LipNet: End-to-End Sentence-level Lipreading
Lipreading is the task of decoding text from the movement of a speaker's mouth.
Ranked #5 on
Lipreading
on GRID corpus (mixed-speech)
Multi-Objective Deep Reinforcement Learning
We propose Deep Optimistic Linear Support Learning (DOL) to solve high-dimensional multi-objective decision problems where the relative importances of the objectives are not known a priori.
Multi-Objective Reinforcement Learning
reinforcement-learning
Alternating Optimisation and Quadrature for Robust Control
ALOQ is robust to the presence of significant rare events, which may not be observable under random sampling, but play a substantial role in determining the optimal policy.
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility.
Probably Approximately Correct Greedy Maximization with Efficient Bounds on Information Gain for Sensor Selection
Submodular function maximization finds application in a variety of real-world decision-making problems.
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks
We propose deep distributed recurrent Q-networks (DDRQN), which enable teams of agents to learn to solve communication-based coordination tasks.
Copeland Dueling Bandits
A version of the dueling bandit problem is addressed in which a Condorcet winner may not exist.
Incremental Clustering and Expansion for Faster Optimal Planning in Dec-POMDPs
We provide theoretical guarantees that, when a suitable heuristic is used, both incremental clustering and incremental expansion yield algorithms that are both complete and search equivalent.
A Survey of Multi-Objective Sequential Decision-Making
Using this taxonomy, we survey the literature on multi-objective methods for planning and learning.
Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problem
This paper proposes a new method for the K-armed dueling bandit problem, a variation on the regular K-armed bandit problem that offers only relative feedback about pairs of arms.
Exploiting Agent and Type Independence in Collaborative Graphical Bayesian Games
Such problems can be modeled as collaborative Bayesian games in which each agent receives private information in the form of its type.
