Search Results for author:
Found 85 papers, 14 papers with code
Simple Ingredients for Offline Reinforcement Learning
Offline reinforcement learning algorithms have proven effective on datasets highly connected to the target downstream task.
Reinforcement Learning with Options and State Representation
The current thesis aims to explore the reinforcement learning field and build on existing methods to produce improved ones to tackle the problem of learning in high-dimensional and complex environments.
Layered State Discovery for Incremental Autonomous Exploration
We leverage these results to design Layered Autonomous Exploration (LAE), a novel algorithm for AX that attains a sample complexity of $\tilde{\mathcal{O}}(LS^{\rightarrow}_{L(1+\epsilon)}\Gamma_{L(1+\epsilon)} A \ln^{12}(S^{\rightarrow}_{L(1+\epsilon)})/\epsilon^2)$, where $S^{\rightarrow}_{L(1+\epsilon)}$ is the number of states that are incrementally $L(1+\epsilon)$-controllable, $A$ is the number of actions, and $\Gamma_{L(1+\epsilon)}$ is the branching factor of the transitions over such states.
Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming.
On the Complexity of Representation Learning in Contextual Linear Bandits
In contextual linear bandits, the reward function is assumed to be a linear combination of an unknown reward vector and a given embedding of context-arm pairs.
Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler
We design a novel algorithmic template, Weak Labeler Active Cover (WL-AC), that is able to robustly leverage the lower quality weak labelers to reduce the query complexity while retaining the desired level of accuracy.
Scalable Representation Learning in Linear Contextual Bandits with Constant Regret Guarantees
We study the problem of representation learning in stochastic contextual linear bandits.
Contextual bandits with concave rewards, and an application to fair ranking
We consider Contextual Bandits with Concave Rewards (CBCR), a multi-objective bandit problem where the desired trade-off between the rewards is defined by a known concave objective function, and the reward vector depends on an observed stochastic context.
Reaching Goals is Hard: Settling the Sample Complexity of the Stochastic Shortest Path
We also initiate the study of learning $\epsilon$-optimal policies without access to a generative model (i. e., the so-called best-policy identification problem), and show that sample-efficient learning is impossible in general.
Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies
We consider infinite-horizon discounted Markov decision processes and study the convergence rates of the natural policy gradient (NPG) and the Q-NPG methods with the log-linear policy class.
Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RL
In reinforcement learning, the graph Laplacian has proved to be a valuable tool in the task-agnostic setting, with applications ranging from skill discovery to reward shaping.
Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
In this work, we propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL.
Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times
Computing a Gaussian process (GP) posterior has a computational cost cubical in the number of historical points.
Top $K$ Ranking for Multi-Armed Bandit with Noisy Evaluations
We consider a multi-armed bandit setting where, at the beginning of each round, the learner receives noisy independent, and possibly biased, \emph{evaluations} of the true reward of each arm and it selects $K$ arms with the objective of accumulating as much reward as possible over $T$ rounds.
Differentially Private Exploration in Reinforcement Learning with Linear Representation
We first consider the setting of linear-mixture MDPs (Ayoub et al., 2020) (a. k. a.\ model-based setting) and provide an unified framework for analyzing joint and local differential private (DP) exploration.
Adaptive Multi-Goal Exploration
We introduce a generic strategy for provably efficient multi-goal exploration.
Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching
Learning meaningful behaviors in the absence of reward is a difficult problem in reinforcement learning.
Reinforcement Learning in Linear MDPs: Constant Regret and Representation Selection
We study the role of the representation of state-action value functions in regret minimization in finite-horizon Markov Decision Processes (MDPs) with linear structure.
A general sample complexity analysis of vanilla policy gradient
We then instantiate our theorems in different settings, where we both recover existing results and obtain improved sample complexity, e. g., $\widetilde{\mathcal{O}}(\epsilon^{-3})$ sample complexity for the convergence to the global optimum for Fisher-non-degenerated parametrized policies.
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control.
A Fully Problem-Dependent Regret Lower Bound for Finite-Horizon MDPs
We derive a novel asymptotic problem-dependent lower-bound for regret minimization in finite-horizon tabular Markov Decision Processes (MDPs).
A Reduction-Based Framework for Conservative Bandits and Reinforcement Learning
We also obtain a new upper bound for conservative low-rank MDP.
Exploration-Driven Representation Learning in Reinforcement Learning
In this work, we introduce a method that explicitly couples representation learning with exploration when the agent is not provided with a uniform prior over the state space.
Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret
We study the problem of learning in the stochastic shortest path (SSP) setting, where an agent seeks to minimize the expected cost accumulated before reaching a goal state.
Leveraging Good Representations in Linear Contextual Bandits
We show that the regret is indeed never worse than the regret obtained by running LinUCB on the best representation (up to a $\ln M$ factor).
Reinforcement Learning with Prototypical Representations
Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations.
Meta-Reinforcement Learning With Informed Policy Regularization
Meta-reinforcement learning aims at finding a policy able to generalize to new environments.
Improved Sample Complexity for Incremental Autonomous Exploration in MDPs
We investigate the exploration of an unknown environment when no reward function is provided.
An Asymptotically Optimal Primal-Dual Incremental Algorithm for Contextual Linear Bandits
Finally, we remove forced exploration and build on confidence intervals of the optimization problem to encourage a minimum level of exploration that is better adapted to the problem structure.
Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration
There has been growing progress on theoretical analyses for provably efficient learning in MDPs with linear function approximation, but much of the existing work has made strong assumptions to enable exploration by conventional exploration frameworks.
A Provably Efficient Sample Collection Strategy for Reinforcement Learning
One of the challenges in online reinforcement learning (RL) is that the agent needs to trade off the exploration of the environment and the exploitation of the samples to optimize its behavior.
Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxation
We study the exploration-exploitation dilemma in the linear quadratic regulator (LQR) setting.
Improved Analysis of UCRL2 with Empirical Bernstein Inequality
We consider the problem of exploration-exploitation in communicating Markov Decision Processes.
A Novel Confidence-Based Algorithm for Structured Bandits
We study finite-armed stochastic bandits where the rewards of each arm might be correlated to those of other arms.
Meta-learning with Stochastic Linear Bandits
The goal is to select a learning algorithm which works well on average over a class of bandits tasks, that are sampled from a task-distribution.
Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
We test the performance of our algorithm in a variety of environments where tasks may vary within each episode.
Active Model Estimation in Markov Decision Processes
Using a number of simple domains with heterogeneous noise in their transitions, we show that our heuristic-based algorithm outperforms both our original algorithm and the maximum entropy algorithm in the small sample regime, while achieving similar asymptotic performance as that of the original algorithm.
Learning Near Optimal Policies with Low Inherent Bellman Error
This has two important consequences: 1) it shows that exploration is possible using only \emph{batch assumptions} with an algorithm that achieves the optimal statistical rate for the setting we consider, which is more general than prior work on low-rank MDPs 2) the lack of closedness (measured by the inherent Bellman error) is only amplified by $\sqrt{d_t}$ despite working in the online setting.
Near-linear Time Gaussian Process Optimization with Adaptive Batching and Resparsification
Gaussian processes (GP) are one of the most successful frameworks to model uncertainty.
Adversarial Attacks on Linear Contextual Bandits
In many of these domains, malicious agents may have incentives to attack the bandit algorithm to induce it to perform a desired behavior.
Conservative Exploration in Reinforcement Learning
While learning in an unknown Markov Decision Process (MDP), an agent should trade off exploration to discover new information about the MDP, and exploitation of the current knowledge to maximize the reward.
Improved Algorithms for Conservative Exploration in Bandits
In this case, it is desirable to deploy online learning algorithms (e. g., a multi-armed bandit algorithm) that interact with the system to learn a better/optimal policy under the constraint that during the learning process the performance is almost never worse than the performance of the baseline itself.
Concentration Inequalities for Multinoulli Random Variables
We investigate concentration inequalities for Dirichlet and Multinomial random variables.
No-Regret Exploration in Goal-Oriented Reinforcement Learning
Many popular reinforcement learning problems (e. g., navigation in a maze, some Atari games, mountain car) are instances of the episodic setting under its stochastic shortest path (SSP) formulation, where an agent has to achieve a goal state while minimizing the cumulative cost.
Limiting Extrapolation in Linear Approximate Value Iteration
We prove that if the features at any state can be represented as a convex combination of features at the anchor points, then errors are propagated linearly over iterations (instead of exponentially) and our method achieves a polynomial sample complexity bound in the horizon and the number of anchor points.
Exploration Bonus for Regret Minimization in Discrete and Continuous Average Reward MDPs
The exploration bonus is an effective approach to manage the exploration-exploitation trade-off in Markov Decision Processes (MDPs).
Regret Bounds for Learning State Representations in Reinforcement Learning
We consider the problem of online reinforcement learning when several state representations (mapping histories to a discrete state space) are available to the learning agent.
Frequentist Regret Bounds for Randomized Least-Squares Value Iteration
We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning (RL).
A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning
While centralized reinforcement learning methods can optimally solve small MAC instances, they do not scale to large problems and they fail to generalize to scenarios different from those seen during training.
Multi-agent Reinforcement Learning
reinforcement-learning
+4
Word-order biases in deep-agent emergent communication
We train models to communicate about paths in a simple gridworld, using miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies.
Gaussian Process Optimization with Adaptive Sketching: Scalable and No Regret
Moreover, we show that our procedure selects at most $\tilde{O}(d_{eff})$ points, where $d_{eff}$ is the effective dimension of the explored space, which is typically much smaller than both $d$ and $t$.
Active Exploration in Markov Decision Processes
As the noise level is initially unknown, we need to trade off the exploration of the environment to estimate the noise and the exploitation of these estimates to compute a policy maximizing the accuracy of the mean predictions.
Exploration Bonus for Regret Minimization in Undiscounted Discrete and Continuous Markov Decision Processes
We introduce and analyse two algorithms for exploration-exploitation in discrete and continuous Markov Decision Processes (MDPs) based on exploration bonuses.
Fighting Boredom in Recommender Systems with Linear Reinforcement Learning
A common assumption in recommender systems (RS) is the existence of a best fixed recommendation strategy.
Rotting bandits are not harder than stochastic ones
In stochastic multi-armed bandits, the reward distribution of each arm is assumed to be stationary.
Near Optimal Exploration-Exploitation in Non-Communicating Markov Decision Processes
While designing the state space of an MDP, it is common to include states that are transient or not reachable by any policy (e. g., in mountain car, the product space of speed and position contains configurations that are not physically reachable).
Improved Regret Bounds for Thompson Sampling in Linear Quadratic Control Problems
Thompson sampling (TS) is an effective approach to trade off exploration and exploration in reinforcement learning.
Improved large-scale graph learning through ridge spectral sparsification
By constructing a spectrally-similar graph, we are able to bound the error induced by the sparsification for a variety of downstream tasks (e. g., SSL).
Distributed Adaptive Sampling for Kernel Matrix Approximation
In this paper, we introduce SQUEAK, a new algorithm for kernel approximation based on RLS sampling that sequentially processes the dataset, storing a dictionary which creates accurate kernel matrix approximations with a number of points that only depends on the effective dimension $d_{eff}(\gamma)$ of the dataset.
Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning
We introduce SCAL, an algorithm designed to perform efficient exploration-exploitation in any unknown weakly-communicating Markov decision process (MDP) for which an upper bound $c$ on the span of the optimal bias function is known.
Regret Minimization in MDPs with Options without Prior Knowledge
The option framework integrates temporal abstraction into the reinforcement learning model through the introduction of macro-actions (i. e., options).
Efficient Second-Order Online Kernel Learning with Adaptive Embedding
The embedded space is continuously updated to guarantee that the embedding remains accurate, and we show that the per-step cost only grows with the effective dimension of the problem and not with $T$.
Second-Order Kernel Online Convex Optimization with Adaptive Sketching
First-order KOCO methods such as functional gradient descent require only $\mathcal{O}(t)$ time and space per iteration, and, when the only information on the losses is their convexity, achieve a minimax optimal $\mathcal{O}(\sqrt{T})$ regret.
Experimental results : Reinforcement Learning of POMDPs using Spectral Methods
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods.
Thompson Sampling for Linear-Quadratic Control Problems
Despite the empirical and theoretical success in a wide range of problems from multi-armed bandit to linear bandit, we show that when studying the frequentist regret TS in control problems, we need to trade-off the frequency of sampling optimistic parameters and the frequency of switches in the control policy.
Exploration--Exploitation in MDPs with Options
While a large body of empirical results show that temporally-extended actions and options may significantly affect the learning performance of an agent, the theoretical understanding of how and when options can be beneficial in online reinforcement learning is relatively limited.
Active Learning for Accurate Estimation of Linear Models
We explore the sequential decision making problem where the goal is to estimate uniformly well a number of linear models, given a shared budget of random contexts independently sampled from a known distribution.
Linear Thompson Sampling Revisited
We derive an alternative proof for the regret of Thompson sampling (\ts) in the stochastic linear bandit setting.
Reinforcement Learning in Rich-Observation MDPs using Spectral Methods
We derive finite-time regret bounds for our algorithm with a weak dependence on the dimensionality of the observed space.
Analysis of Kelner and Levin graph sparsification algorithm for a streaming setting
We derive a new proof to show that the incremental resparsification algorithm proposed by Kelner and Levin (2013) produces a spectral sparsifier in high probability.
Open Problem: Approximate Planning of POMDPs in the class of Memoryless Policies
Generally in RL, one can assume a generative model, e. g. graphical models, for the environment, and then the task for the RL agent is to learn the model parameters and find the optimal strategy based on these learnt parameters.
Reinforcement Learning of POMDPs using Spectral Methods
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods.
Incremental Spectral Sparsification for Large-Scale Graph-Based Semi-Supervised Learning
While the harmonic function solution performs well in many semi-supervised learning (SSL) tasks, it is known to scale poorly with the number of samples.
Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits
If the variance of the distributions were known, one could design an optimal sampling strategy by collecting a number of independent samples per distribution that is proportional to their variance.
Exploiting easy data in online optimization
We consider the problem of online optimization, where a learner chooses a decision from a given decision set and suffers some loss associated with the decision and the state of the environment.
Sparse Multi-Task Reinforcement Learning
This is equivalent to assuming that the weight vectors of the task value functions are \textit{jointly sparse}, i. e., the set of their non-zero components is small and it is shared across tasks.
Best-Arm Identification in Linear Bandits
We study the best-arm identification problem in linear bandit, where the rewards of the arms depend linearly on an unknown parameter $\theta^*$ and the objective is to return the arm with the largest reward.
Online Stochastic Optimization under Correlated Bandit Feedback
In this paper we consider the problem of online stochastic optimization of a locally smooth function under bandit feedback.
Sequential Transfer in Multi-armed Bandit with Finite Set of Models
Learning from prior tasks and transferring that experience to improve future performance is critical for building lifelong learning agents.
Regret Bounds for Reinforcement Learning with Policy Advice
In some reinforcement learning problems an agent may be provided with a set of input policies, perhaps learned from prior experience or provided by advisors.
Risk-Aversion in Multi-armed Bandits
In stochastic multi--armed bandits the objective is to solve the exploration--exploitation dilemma and ultimately maximize the expected reward.
Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence
We study the problem of identifying the best arm(s) in the stochastic multi-armed bandit setting.
Multi-Bandit Best Arm Identification
We first propose an algorithm called Gap-based Exploration (GapE) that focuses on the arms whose mean is close to the mean of the best arm in the same bandit (i. e., small gap).
Transfer from Multiple MDPs
Transfer reinforcement learning (RL) methods leverage on the experience collected on a set of source tasks to speed-up RL algorithms.
LSTD with Random Projections
We provide a thorough theoretical analysis of the LSTD with random projections and derive performance bounds for the resulting algorithm.
