Search Results for author:
Found 45 papers, 21 papers with code
Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios
To our knowledge, this is the first application of a combined imitation and reinforcement learning approach in autonomous driving that utilizes large amounts of real-world human driving data.
Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
The potential of offline reinforcement learning (RL) is that high-capacity models trained on large, heterogeneous datasets can lead to agents that generalize broadly, analogously to similar advances in vision and NLP.
Oracle Inequalities for Model Selection in Offline Reinforcement Learning
We propose the first model selection algorithm for offline RL that achieves minimax rate-optimal oracle inequalities up to logarithmic factors.
Model Selection in Batch Policy Optimization
We formalize the problem in the contextual bandit setting with linear model classes by identifying three sources of error that any model selection algorithm should optimally trade-off in order to be competitive: (1) approximation error, (2) statistical complexity, and (3) coverage.
DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization
In this paper, we discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the offline deep RL setting, leading to poor generalization and degenerate feature representations.
Coupled Gradient Estimators for Discrete Latent Variables
Training models with discrete latent variables is challenging due to the high variance of unbiased gradient estimators.
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics.
Benchmarks for Deep Off-Policy Evaluation
Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making.
Offline Policy Selection under Uncertainty
More importantly, we show how the belief distribution estimated by BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric, and we empirically demonstrate that this selection procedure significantly outperforms existing approaches, such as ranking policies according to mean or high-confidence lower bound value estimates.
RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget, thus making RL research both more systematic and more accessible across the community.
DisARM: An Antithetic Gradient Estimator for Binary Latent Variables
Applying antithetic sampling over the augmenting variables yields a relatively low-variance and unbiased estimator applicable to any model with binary latent variables.
Conservative Q-Learning for Offline Reinforcement Learning
We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection.
Model Based Reinforcement Learning for Atari
We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting.
D4RL: Datasets for Deep Data-Driven Reinforcement Learning
In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL.
Meta-Learning without Memorization
If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes.
Behavior Regularized Offline Reinforcement Learning
In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment.
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables.
Energy-Inspired Models: Learning with Sampler-Induced Distributions
Motivated by this, we consider the sampler-induced distribution as the model of interest and maximize the likelihood of this model.
Reinforcement Learning Driven Heuristic Optimization
In this paper, we introduce a novel framework to generate better initial solutions for heuristic algorithms using reinforcement learning (RL), named RLHO.
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator.
On Variational Bounds of Mutual Information
Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging.
Guided Evolutionary Strategies: Escaping the curse of dimensionality in random search
This arises when an approximate gradient is easier to compute than the full gradient (e. g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e. g. in certain reinforcement learning applications or training networks with discrete variables).
Understanding Posterior Collapse in Generative Latent Variable Models
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational distribution closely matches the uninformative prior for a subset of latent variables.
Revisiting Auxiliary Latent Variables in Generative Models
The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.
Model-Based Reinforcement Learning for Atari
We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting.
Ranked #8 on
Atari Games 100k
on Atari 100k
Learning to Walk via Deep Reinforcement Learning
In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies.
Soft Actor-Critic Algorithms and Applications
A fork of OpenAI Baselines, implementations of reinforcement learning algorithms
The Laplacian in RL: Learning Representations with Efficient Approximations
In this paper, we present a fully general and scalable method for approximating the eigenvectors of the Laplacian in a model-free RL context.
Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives
Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
Integrating model-free and model-based approaches in reinforcement learning has the potential to achieve the high performance of model-free algorithms with low sample complexity.
Guided evolutionary strategies: Augmenting random search with surrogate gradients
We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search.
Smoothed Action Value Functions for Learning Gaussian Policies
State-action value functions (i. e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning.
The Mirage of Action-Dependent Baselines in Reinforcement Learning
Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance.
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical.
Ranked #1 on
Multi-Armed Bandits
on Mushroom
Learning Gaussian Policies from Smoothed Action Value Functions
We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value used in SARSA.
Combining Model-based and Model-free RL via Multi-step Control Variates
Model-free deep reinforcement learning algorithms are able to successfully solve a wide range of continuous control tasks, but typically require many on-policy samples to achieve good performance.
An online sequence-to-sequence model for noisy speech recognition
This is because the models require that the entirety of the input sequence be available at the beginning of inference, an assumption that is not valid for instantaneous speech recognition.
Filtering Variational Objectives
When used as a surrogate objective for maximum likelihood estimation in latent variable models, the evidence lower bound (ELBO) produces state-of-the-art results.
Learning Hard Alignments with Variational Inference
There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition.
Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spotting
Finally, the max-pooling loss trained LSTM initialized with a cross-entropy pre-trained network shows the best performance, which yields $67. 6\%$ relative reduction compared to baseline feed-forward DNN in Area Under the Curve (AUC) measure.
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
Learning in models with discrete latent variables is challenging due to high variance gradient estimators.
Particle Value Functions
The policy gradients of the expected return objective can react slowly to rare rewards.
Regularizing Neural Networks by Penalizing Confident Output Distributions
We systematically explore regularizing neural networks by penalizing low entropy output distributions.
Compacting Neural Network Classifiers via Dropout Training
We introduce dropout compaction, a novel method for training feed-forward neural networks which realizes the performance gains of training a large model with dropout regularization, yet extracts a compact neural network for run-time efficiency.
