Search Results for author:
Found 79 papers, 19 papers with code
Multi-turn Reinforcement Learning from Preference Human Feedback
Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning Large Language Models (LLMs) with human preferences, allowing LLMs to demonstrate remarkable abilities in various tasks.
Understanding the performance gap between online and offline alignment algorithms
However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF.
Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), resolving an open question of Zhang et al. (2023).
Distributional Reinforcement Learning
reinforcement-learning
+1
Off-policy Distributional Q($λ$): Distributional RL without Importance Sampling
We introduce off-policy distributional Q($\lambda$), a new addition to the family of off-policy distributional evaluation algorithms.
Generalized Preference Optimization: A Unified Approach to Offline Alignment
The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss.
A General Theoretical Paradigm to Understand Learning from Human Preferences
In particular we derive a new general objective called $\Psi$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations.
Local and adaptive mirror descents in extensive-form games
We study how to learn $\epsilon$-optimal strategies in zero-sum imperfect information games (IIG) with trajectory feedback.
DoMo-AC: Doubly Multi-step Off-policy Actor-Critic Algorithm
Multi-step learning applies lookahead over multiple time steps and has proved valuable in policy evaluation settings.
Towards a Better Understanding of Representation Dynamics under TD-learning
Complementary to prior work, we provide a set of analysis that sheds further light on the representation dynamics under TD-learning.
VA-learning as a more efficient alternative to Q-learning
In reinforcement learning, the advantage function is critical for policy improvement, but is often extracted from a learned Q-function.
The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation
We study the problem of temporal-difference-based policy evaluation in reinforcement learning.
Distributional Reinforcement Learning
reinforcement-learning
Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice
Mirror descent value iteration (MDVI), an abstraction of Kullback-Leibler (KL) and entropy-regularized reinforcement learning (RL), has served as the basis for recent high-performing practical RL algorithms.
An Analysis of Quantile Temporal-Difference Learning
We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning.
Distributional Reinforcement Learning
reinforcement-learning
+1
Adapting to game trees in zero-sum imperfect information games
Imperfect information games (IIG) are games in which each player only partially observes the current game state.
Understanding Self-Predictive Learning for Reinforcement Learning
We identify that a faster paced optimization of the predictor and semi-gradient updates on the representation, are crucial to preventing the representation collapse.
Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments
In this work, we study a natural solution derived from structural causal models of the world: Our key idea is to learn representations of the future that capture precisely the unpredictable aspects of each outcome -- which we use as additional input for predictions, such that intrinsic rewards only reflect the predictable aspects of world dynamics.
The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning
We study the multi-step off-policy learning approach to distributional RL.
Distributional Reinforcement Learning
quantile regression
+2
Generalised Policy Improvement with Geometric Policy Composition
We introduce a method for policy improvement that interpolates between the greedy approach of value-based reinforcement learning (RL) and the full planning approach typical of model-based RL.
BYOL-Explore: Exploration by Bootstrapped Prediction
We present BYOL-Explore, a conceptually simple yet general approach for curiosity-driven exploration in visually-complex environments.
KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal
In this work, we consider and analyze the sample complexity of model-free reinforcement learning with a generative model.
Marginalized Operators for Off-policy Reinforcement Learning
We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases.
Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions.
Taylor Expansion of Discount Factors
In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective.
Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall
We study the problem of learning a Nash equilibrium (NE) in an imperfect information game (IIG) through self-play.
Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint
Mean-field Games (MFGs) are a continuous approximation of many-agent RL.
Revisiting Peng's Q($λ$) for Modern Reinforcement Learning
These results indicate that Peng's Q($\lambda$), which was thought to be unsafe, is a theoretically-sound and practically effective algorithm.
Large-Scale Representation Learning on Graphs via Bootstrapping
To address these challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method that learns by predicting alternative augmentations of the input.
Geometric Entropic Exploration
Exploration is essential for solving complex Reinforcement Learning (RL) tasks.
Counterfactual Credit Assignment in Model-Free Reinforcement Learning
Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards.
The Advantage Regret-Matching Actor-Critic
In this paper, we describe a general model-free RL method for no-regret learning based on repeated reconsideration of past behavior.
Monte-Carlo Tree Search as Regularized Policy Optimization
The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence.
Bootstrap your own latent: A new approach to self-supervised Learning
From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
Ranked #2 on
Self-Supervised Person Re-Identification
on SYSU-30k
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
These latent embeddings are themselves trained to be predictive of the aforementioned representations.
Leverage the Average: an Analysis of KL Regularization in RL
Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance.
Taylor Expansion Policy Optimization
In this work, we investigate the application of Taylor expansions in reinforcement learning.
Adaptive Trade-Offs in Off-Policy Learning
A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms.
Conditional Importance Sampling for Off-Policy Learning
We theoretically analyse this space, and concretely investigate several algorithms that arise from this framework.
Towards Consistent Performance on Atari using Expert Demonstrations
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games.
Statistics and Samples in Distributional Reinforcement Learning
We present a unifying framework for designing and analysing distributional reinforcement learning (DRL) algorithms in terms of recursively estimating statistics of the return distribution.
Distributional Reinforcement Learning
reinforcement-learning
+1
Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
In this paper we extend the SFs & GPI framework in two ways.
Optimistic optimization of a Brownian
Given $W$, our goal is to return an $\epsilon$-approximation of its maximum using the smallest possible number of function evaluations, the sample complexity of the algorithm.
Universal Successor Features Approximators
We focus on one aspect in particular, namely the ability to generalise to unseen tasks.
Neural Predictive Belief Representations
In partially observable domains it is important for the representation to encode a belief state, a sufficient statistic of the observations seen so far.
Implicit Quantile Networks for Distributional Reinforcement Learning
In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN.
Ranked #1 on
Atari Games
on Atari 2600 Freeway
Autoregressive Quantile Networks for Generative Modeling
We introduce autoregressive implicit quantile networks (AIQN), a fundamentally different approach to generative modeling than those commonly used, that implicitly captures the distribution using quantile regression.
Observe and Look Further: Achieving Consistent Performance on Atari
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games.
An Analysis of Categorical Distributional Reinforcement Learning
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance.
Distributional Reinforcement Learning
reinforcement-learning
+1
Learning to Search with MCTSnets
They are most typically solved by tree search algorithms that simulate ahead into the future, evaluate future states, and back-up those evaluations to the root of a search tree.
Distributional Reinforcement Learning with Quantile Regression
In this paper, we build on recent work advocating a distributional approach to reinforcement learning in which the distribution over returns is modeled explicitly instead of only estimating the mean.
Ranked #1 on
Atari Games
on Atari 2600 Pong
A Distributional Perspective on Reinforcement Learning
We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning.
Ranked #4 on
Atari Games
on Atari 2600 HERO
Observational Learning by Reinforcement Learning
Observational learning is a type of learning that occurs as a function of observing, retaining and possibly replicating or imitating the behaviour of another agent.
The Cramer Distance as a Solution to Biased Wasserstein Gradients
We show that the Cram\'er distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences.
Minimax Regret Bounds for Reinforcement Learning
We consider the problem of provably optimal exploration in reinforcement learning for finite horizon MDPs.
Successor Features for Transfer in Reinforcement Learning
Transfer in reinforcement learning refers to the notion that generalization should occur not only within a task but also across tasks.
Safe and Efficient Off-Policy Reinforcement Learning
In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning.
Increasing the Action Gap: New Operators for Reinforcement Learning
Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator.
Ranked #1 on
Atari Games
on Atari 2600 Elevator Action
Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits
If the variance of the distributions were known, one could design an optimal sampling strategy by collecting a number of independent samples per distribution that is proportional to their variance.
Best-Arm Identification in Linear Bandits
We study the best-arm identification problem in linear bandit, where the rewards of the arms depend linearly on an unknown parameter $\theta^*$ and the objective is to return the arm with the largest reward.
Fast gradient descent for drifting least squares regression, with application to bandits
In the case when strong convexity in the regression problem is guaranteed, we provide bounds on the error both in expectation and high probability (the latter is often needed to provide theoretical guarantees for higher level algorithms), despite the drifting least squares solution.
Concentration bounds for temporal difference learning with linear function approximation: The case of batch data and uniform sampling
We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm.
Adaptive Stratified Sampling for Monte-Carlo integration of Differentiable functions
We consider the problem of adaptive stratified sampling for Monte Carlo integration of a differentiable function given a finite number of evaluations to the function.
Bandit Algorithms boost Brain Computer Interfaces for motor-task selection of a brain-controlled button
A brain-computer interface (BCI) allows users to “communicate” with a computer without using their muscles.
Risk-Aversion in Multi-armed Bandits
In stochastic multi--armed bandits the objective is to solve the exploration--exploitation dilemma and ultimately maximize the expected reward.
Thompson Sampling: An Asymptotically Optimal Finite Time Analysis
The question of the optimality of Thompson Sampling for solving the stochastic multi-armed bandit problem had been open since 1933.
Selecting the State-Representation in Reinforcement Learning
Without knowing neither which of the models is the correct one, nor what are the probabilistic characteristics of the resulting MDP, it is required to obtain as much reward as the optimal policy for the correct model (or for the best of the correct models, if there are several).
Finite Time Analysis of Stratified Sampling for Monte Carlo
We consider the problem of stratified sampling for Monte-Carlo integration.
Speedy Q-Learning
We introduce a new convergent variant of Q-learning, called speedy Q-learning, to address the problem of slow convergence in the standard form of the Q-learning algorithm.
Sparse Recovery with Brownian Sensing
We consider the problem of recovering the parameter alpha in R^K of a sparse function f, i. e. the number of non-zero entries of alpha is small compared to the number K of features, given noisy evaluations of f at a set of well-chosen sampling points.
LSTD with Random Projections
We provide a thorough theoretical analysis of the LSTD with random projections and derive performance bounds for the resulting algorithm.
Error Propagation for Approximate Policy and Value Iteration
We address the question of how the approximation error/Bellman residual at each iteration of the Approximate Policy/Value Iteration algorithms influences the quality of the resulted policy.
Scrambled Objects for Least-Squares Regression
We consider least-squares regression using a randomly generated subspace G_P\subset F of finite dimension P, where F is a function space of infinite dimension, e. g.~L_2([0, 1]^d).
Compressed Least-Squares Regression
We consider the problem of learning, from K input data, a regression function in a function space of high dimension N using projections onto a random subspace of lower dimension M. From any linear approximation algorithm using empirical risk minimization (possibly penalized), we provide bounds on the excess risk of the estimate computed in the projected subspace (compressed domain) in terms of the excess risk of the estimate built in the high-dimensional space (initial domain).
Sensitivity analysis in HMMs with application to likelihood maximization
We derive an IPA estimator for the gradient of the log-likelihood, which may be used in a gradient method for the purpose of likelihood maximization.
Algorithms for Infinitely Many-Armed Bandits
We consider multi-armed bandit problems where the number of arms is larger than the possible number of experiments.
Particle Filter-based Policy Gradient in POMDPs
Our setting is a Partially Observable Markov Decision Process with continuous state, observation and action spaces.
Online Optimization in X-Armed Bandits
We consider a generalization of stochastic bandit problems where the set of arms, X, is allowed to be a generic topological space.
Fitted Q-iteration in continuous action-space MDPs
We consider continuous state, continuous action batch reinforcement learning where the goal is to learn a good policy from a sufficiently rich trajectory generated by another policy.
