Search Results for author:
Found 58 papers, 28 papers with code
Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs
As a corollary to this work, we find that REINFORCE's baseline parameter plays an important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving off-policy performance.
Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown.
Distributional Reinforcement Learning
reinforcement-learning
+2
Controlling Large Language Model Agents with Entropic Activation Steering
But how do LLM agents explore, and how can we control their exploratory behaviors?
A Distributional Analogue to the Successor Representation
This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process.
Distributional Reinforcement Learning
Model-based Reinforcement Learning
+2
Learning and Controlling Silicon Dopant Transitions in Graphene using Scanning Transmission Electron Microscopy
We introduce a machine learning approach to determine the transition dynamics of silicon atoms on a single layer of carbon atoms, when stimulated by the electron beam of a scanning transmission electron microscope (STEM).
Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy.
Bootstrapped Representations in Reinforcement Learning
In this paper, we address this gap and provide a theoretical characterization of the state representation learnt by temporal difference learning (Sutton, 1988).
The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation
We study the problem of temporal-difference-based policy evaluation in reinforcement learning.
Distributional Reinforcement Learning
reinforcement-learning
+1
Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks
Combined with a suitable off-policy learning rule, the result is a representation learning algorithm that can be understood as extending Mahadevan & Maggioni (2007)'s proto-value functions to deep reinforcement learning -- accordingly, we call the resulting object proto-value networks.
An Analysis of Quantile Temporal-Difference Learning
We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning.
Distributional Reinforcement Learning
reinforcement-learning
+2
A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspaces
In this paper, we derive an algorithm that learns a principal subspace from sample entries, can be applied when the approximate subspace is represented by a neural network, and hence can be scaled to datasets with an effectively infinite number of rows and columns.
The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning
We study the multi-step off-policy learning approach to distributional RL.
Distributional Reinforcement Learning
quantile regression
+2
Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress
To address these issues, we present reincarnating RL as an alternative workflow or class of problem settings, where prior computational work (e. g., learned policies) is reused or transferred between design iterations of an RL agent, or from one RL agent to another.
Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning
We demonstrate the effectiveness of such an algorithm in a synthetic control problem.
On the Generalization of Representations in Reinforcement Learning
We complement our theoretical results with an empirical survey of classic representation learning methods from the literature and results on the Arcade Learning Environment, and find that the generalization behaviour of learned representations is well-explained by their effective dimension.
On Bonus-Based Exploration Methods in the Arcade Learning Environment
Research on exploration in reinforcement learning, as applied to Atari 2600 game-playing, has emphasized tackling difficult exploration problems such as Montezuma's Revenge (Bellemare et al., 2016).
Deep Reinforcement Learning at the Edge of the Statistical Precipice
Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs.
Metrics and continuity in reinforcement learning
In most practical applications of reinforcement learning, it is untenable to maintain direct estimates for individual states; in continuous-state systems, it is impossible.
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states.
The Importance of Pessimism in Fixed-Dataset Policy Optimization
To avoid this, algorithms can follow the pessimism principle, which states that we should choose the policy which acts optimally in the worst possible world.
Representations for Stable Off-Policy Reinforcement Learning
Reinforcement learning with function approximation can be unstable and even divergent, especially when combined with off-policy learning and Bellman updates.
The Value-Improvement Path: Towards Better Representations for Reinforcement Learning
To test our hypothesis empirically, we augmented a standard deep RL agent with an auxiliary task of learning the value-improvement path.
A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms
We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes.
Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces
Despite the wealth of research into provably efficient reinforcement learning algorithms, most works focus on tabular representation and thus struggle to handle exponentially or infinitely large state-action spaces.
On Catastrophic Interference in Atari 2600 Games
Our study provides a clear empirical link between catastrophic interference and sample efficiency in reinforcement learning.
On Bonus Based Exploration Methods In The Arcade Learning Environment
Research on exploration in reinforcement learning, as applied to Atari 2600 game-playing, has emphasized tackling difficult exploration problems such as Montezuma's Revenge (Bellemare et al., 2016).
Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction
Empirically, we find that these techniques improve the performance of a baseline deep reinforcement learning agent applied to text-based games.
Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
This paper provides an empirical evaluation of recently developed exploration algorithms within the Arcade Learning Environment (ALE).
DeepMDP: Learning Continuous Latent Space Models for Representation Learning
We show that the optimization of these objectives guarantees (1) the quality of the latent space as a representation of the state space and (2) the quality of the DeepMDP as a model of the environment.
Statistics and Samples in Distributional Reinforcement Learning
We present a unifying framework for designing and analysing distributional reinforcement learning (DRL) algorithms in terms of recursively estimating statistics of the return distribution.
Distributional Reinforcement Learning
reinforcement-learning
+2
Hyperbolic Discounting and Learning over Multiple Horizons
Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process.
Distributional reinforcement learning with linear function approximation
Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited.
Distributional Reinforcement Learning
reinforcement-learning
+2
The Hanabi Challenge: A New Frontier for AI Research
From the early days of computing, games have been important testbeds for studying how well machines can do sophisticated decision making.
The Value Function Polytope in Reinforcement Learning
We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes.
Shaping the Narrative Arc: An Information-Theoretic Approach to Collaborative Dialogue
We consider the problem of designing an artificial agent capable of interacting with humans in collaborative dialogue to produce creative, engaging narratives.
A Geometric Perspective on Optimal Representations for Reinforcement Learning
We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks.
A Comparative Analysis of Expected and Distributional Reinforcement Learning
Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL).
Distributional Reinforcement Learning
reinforcement-learning
+2
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift
We complement our analysis with an empirical evaluation of the two techniques in an off-policy setting on the game Pong from the Atari domain where we find discounted COP-TD to be better behaved in practice than the soft normalization penalty.
An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents
We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous Deep RL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models.
Dopamine: A Research Framework for Deep Reinforcement Learning
Dopamine is a research framework for fast prototyping of reinforcement learning algorithms.
An Introduction to Deep Reinforcement Learning
Deep reinforcement learning is the combination of reinforcement learning (RL) and deep learning.
Approximate Exploration through State Abstraction
Next, we show how a given density model can be related to an abstraction and that the corresponding pseudo-count bonus can act as a substitute in MBIE-EB combined with this abstraction, but may lead to either under- or over-exploration.
Count-Based Exploration with the Successor Representation
In this paper we introduce a simple approach for exploration in reinforcement learning (RL) that allows us to develop theoretically justified algorithms in the tabular case but that is also extendable to settings where function approximation is required.
Ranked #16 on
Atari Games
on Atari 2600 Venture
An Analysis of Categorical Distributional Reinforcement Learning
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance.
Distributional Reinforcement Learning
reinforcement-learning
+2
Distributional Reinforcement Learning with Quantile Regression
In this paper, we build on recent work advocating a distributional approach to reinforcement learning in which the distribution over returns is modeled explicitly instead of only estimating the mean.
Ranked #1 on
Atari Games
on Atari 2600 Pong
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games.
A Distributional Perspective on Reinforcement Learning
We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning.
Ranked #4 on
Atari Games
on Atari 2600 HERO
The Cramer Distance as a Solution to Biased Wasserstein Gradients
We show that the Cram\'er distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences.
Automated Curriculum Learning for Neural Networks
We introduce a method for automatically selecting the path, or syllabus, that a neural network follows through a curriculum so as to maximise learning efficiency.
Count-Based Exploration with Neural Density Models
This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge.
Ranked #9 on
Atari Games
on Atari 2600 Montezuma's Revenge
A Laplacian Framework for Option Discovery in Reinforcement Learning
Representation learning and option discovery are two of the biggest challenges in reinforcement learning (RL).
Safe and Efficient Off-Policy Reinforcement Learning
In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning.
Unifying Count-Based Exploration and Intrinsic Motivation
We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations.
Ranked #10 on
Atari Games
on Atari 2600 Montezuma's Revenge
Q($λ$) with Off-Policy Corrections
We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities.
Increasing the Action Gap: New Operators for Reinforcement Learning
Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator.
Ranked #1 on
Atari Games
on Atari 2600 Elevator Action
Human level control through deep reinforcement learning
We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters.
Ranked #20 on
Atari Games
on Atari 2600 Ice Hockey
Compress and Control
This paper describes a new information-theoretic policy evaluation technique for reinforcement learning.
The Arcade Learning Environment: An Evaluation Platform for General Agents
We illustrate the promise of ALE by developing and benchmarking domain-independent agents designed using well-established AI techniques for both reinforcement learning and planning.
Ranked #1 on
Atari Games
on Atari 2600 Pooyan
