Search Results for author:
Found 92 papers, 27 papers with code
Tuning for the Unknown: Revisiting Evaluation Strategies for Lifelong RL
In this paper we propose a new approach for tuning and evaluating lifelong RL agents where only one percent of the experiment data can be used for hyperparameter tuning.
Investigating the Histogram Loss in Regression
It is becoming increasingly common in regression to train neural networks that model the entire distribution even if only the mean is required for prediction.
What to Do When Your Discrete Optimization Is the Size of a Neural Network?
Oftentimes, machine learning applications using neural networks involve solving discrete optimization problems, such as in pruning, parameter-isolation-based continual learning and training of binary networks.
Compound Returns Reduce Variance in Reinforcement Learning
Multistep returns, such as $n$-step returns and $\lambda$-returns, are commonly used to improve the sample efficiency of reinforcement learning (RL) methods.
When is Offline Policy Selection Sample Efficient for Reinforcement Learning?
As a result, no OPS method can be more sample efficient than OPE in the worst case.
GVFs in the Real World: Making Predictions Online for Water Treatment
In this paper we investigate the use of reinforcement-learning based prediction approaches for a real drinking-water treatment plant.
Measuring and Mitigating Interference in Reinforcement Learning
Lastly, we outline a class of algorithms which we call online-aware that are designed to mitigate interference, and show they do reduce interference according to our measure and that they improve stability and performance in several classic control environments.
Coagent Networks: Generalized and Scaled
However, the coagent framework is not just an alternative to BDL; the two approaches can be blended: BDL can be combined with coagent learning rules to create architectures with the advantages of both approaches.
Empirical Design in Reinforcement Learning
The objective of this document is to provide answers on how we can use our unprecedented compute to do good science in reinforcement learning, as well as stay alert to potential pitfalls in our empirical design.
The In-Sample Softmax for Offline Reinforcement Learning
We highlight a simple fact: it is more straightforward to approximate an in-sample \emph{softmax} using only actions in the dataset.
Asymptotically Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonstationary Environments
In this work, we consider the off-policy policy evaluation problem for contextual bandits and finite horizon reinforcement learning in the nonstationary setting.
Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence
Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickly.
Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning
Off-policy learning from multistep returns is crucial for sample-efficient reinforcement learning, but counteracting off-policy bias without exacerbating variance is challenging.
Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks
We show that by either decomposing the network into independent modules or learning the network in stages, we can make RTRL scale linearly with the number of parameters.
Goal-Space Planning with Subgoal Models
In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL
The performance of reinforcement learning (RL) agents is sensitive to the choice of hyperparameters.
Robust Losses for Learning Value Functions
We start from a formalization of robust losses, then derive sound gradient-based approaches to minimize these losses in both the online off-policy prediction and control settings.
Investigating the Properties of Neural Network Representations in Reinforcement Learning
In this paper we investigate the properties of representations learned by deep reinforcement learning systems.
Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentum
In this work, we show that SGDm under covariate shift with a fixed step-size can be unstable and diverge.
Continual Auxiliary Task Learning
Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems.
A Temporal-Difference Approach to Policy Gradient Estimation
The policy gradient theorem (Sutton et al., 2000) prescribes the usage of a cumulative discounted state distribution under the target policy to approximate the gradient.
An Alternate Policy Gradient Estimator for Softmax Policies
Policy gradient (PG) estimators are ineffective in dealing with softmax policies that are sub-optimally saturated, which refers to the situation when the policy concentrates its probability mass on sub-optimal actions.
Representation Alignment in Neural Networks
We then highlight why alignment between the top singular vectors and the targets can speed up learning and show in a classic synthetic transfer problem that representation alignment correlates with positive and negative transfer to similar and dissimilar tasks.
Structural Credit Assignment in Neural Networks using Reinforcement Learning
In this work, we revisit this approach and investigate if we can leverage other reinforcement learning approaches to improve learning.
Off-Policy Actor-Critic with Emphatic Weightings
A variety of theoretically-sound policy gradient algorithms exist for the on-policy setting due to the policy gradient theorem, which provides a simplified form for the gradient.
Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
Offline reinforcement learning -- learning a policy from a batch of data -- is known to be hard for general MDPs.
Resmax: An Alternative Soft-Greedy Operator for Reinforcement Learning
Soft-greedy operators, namely $\varepsilon$-greedy and softmax, remain a common choice to induce a basic level of exploration for action-value methods in reinforcement learning.
Offline-Online Reinforcement Learning: Extending Batch and Online RL
To enable better performance, we investigate the offline-online setting: The agent has access to a batch of data to train on but is also allowed to learn during the evaluation phase in an online manner.
Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification.
Predictive Representation Learning for Language Modeling
Correlates of secondary information appear in LSTM representations even though they are not part of an \emph{explicitly} supervised prediction task.
A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
Many algorithms have been developed for off-policy value estimation based on the linear mean squared projected Bellman error (MSPBE) and are sound under linear function approximation.
Scalable Online Recurrent Learning Using Columnar Neural Networks
We empirically show that as long as connections between columns are sparse, our method approximates the true gradient well.
Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshop
This report presents the debates, posters, and discussions of the Sim2Real workshop held in conjunction with the 2020 edition of the "Robotics: Science and System" conference.
Towards Safe Policy Improvement for Non-Stationary MDPs
Many real-world sequential decision-making problems involve critical systems with financial risks and human-life risks.
From Language to Language-ish: How Brain-Like is an LSTM's Representation of Nonsensical Language Stimuli?
The representations generated by many models of language (word embeddings, recurrent neural networks and transformers) correlate to brain activity recorded while people read.
Beyond Prioritized Replay: Sampling States in Model-Based RL via Simulated Priorities
The prioritized Experience Replay (ER) method has attracted great attention; however, there is little theoretical understanding of such prioritization strategy and why they help.
Understanding and Mitigating the Limitations of Prioritized Experience Replay
Prioritized Experience Replay (ER) has been empirically shown to improve sample efficiency across many domains and attracted great attention; however, there is little theoretical understanding of why such prioritized sampling helps and its limitations.
Towards a practical measure of interference for reinforcement learning
In this work, we provide a definition of interference for control in reinforcement learning.
Selective Dyna-style Planning Under Limited Model Capacity
We show that heteroscedastic regression can signal predictive uncertainty arising from model inadequacy that is complementary to that which is detected by methods designed for parameter uncertainty, indicating that considering both parameter uncertainty and model inadequacy may be a more promising direction for effective selective planning than either in isolation.
Gradient Temporal-Difference Learning with Regularized Corrections
It is still common to use Q-learning and temporal difference (TD) learning-even though they have divergence issues and sound Gradient TD alternatives exist-because divergence seems rare and they typically perform well.
Learning Causal Models Online
One solution for achieving strong generalization is to incorporate causal structures in the models; such structures constrain learning by ignoring correlations that contradict them.
Hallucinating Value: A Pitfall of Dyna-style Planning with Imperfect Environment Models
Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model.
Optimizing for the Future in Non-Stationary MDPs
Most reinforcement learning methods are based upon the key assumption that the transition dynamics and reward functions are fixed, that is, the underlying Markov decision process is stationary.
Maximizing Information Gain in Partially Observable Environments via Prediction Reward
Information gathering in a partially observable environment can be formulated as a reinforcement learning (RL), problem where the reward depends on the agent's uncertainty.
Training Recurrent Neural Networks Online by Learning Explicit State Variables
Recurrent neural networks (RNNs) allow an agent to construct a state-representation from a stream of experience, which is essential in partially observable problems.
Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
Q-learning suffers from overestimation bias, because it approximates the maximum action value using the maximum estimated action value.
An implicit function learning approach for parametric modal regression
We empirically demonstrate on several synthetic problems that our method (i) can learn multi-valued functions and produce the conditional modes, (ii) scales well to high-dimensional inputs, and (iii) can even be more effective for certain uni-modal problems, particularly for high-frequency functions.
Learning Macroscopic Brain Connectomes via Group-Sparse Factorization
We develop an efficient optimization strategy for this extremely high-dimensional sparse problem, by reducing the number of parameters using a greedy algorithm designed specifically for the problem.
Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online
Recent work has shown that sparse representations -- where only a small percentage of units are active -- can significantly reduce interference.
Is Fast Adaptation All You Need?
Gradient-based meta-learning has proven to be highly effective at learning model initializations, representations, and update rules that allow fast adaptation from a few samples.
Meta-descent for Online, Continual Prediction
This paper investigates different vector step-size adaptation approaches for non-stationary online, continual prediction problems.
Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
The question we tackle in this paper is how to sculpt the stream of experience---how to adapt the learning system's behavior---to optimize the learning of a collection of value functions.
Hill Climbing on Value Estimates for Search-control in Dyna
In this work, we propose to generate such states by using the trajectory obtained from Hill Climbing (HC) the current estimate of the value function.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Importance Resampling for Off-policy Prediction
Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning.
Meta-Learning Representations for Continual Learning
We show that it is possible to learn naturally sparse representations that are more effective for online updating.
Two-Timescale Networks for Nonlinear Value Function Approximation
A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control.
Planning with Expectation Models
In particular, we 1) show that planning with an expectation model is equivalent to planning with a distribution model if the state value function is linear in state features, 2) analyze two common parametrization choices for approximating the expectation: linear and non-linear expectation models, 3) propose a sound model-based policy evaluation algorithm and present its convergence results, and 4) empirically demonstrate the effectiveness of the proposed planning algorithm.
Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
Knowledge distillation is an effective technique that transfers knowledge from a large teacher model to a shallow student.
Supervised autoencoders: Improving generalization performance with unsupervised regularizers
A common strategy to improve generalization has been through the use of regularizers, typically as a norm constraining the parameters.
An Off-policy Policy Gradient Theorem Using Emphatic Weightings
There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradient.
The Barbados 2018 List of Open Issues in Continual Learning
We want to make progress toward artificial general intelligence, namely general-purpose agents that autonomously learn how to competently act in complex environments.
Context-Dependent Upper-Confidence Bounds for Directed Exploration
Directed exploration strategies for reinforcement learning are critical for learning an optimal policy in a minimal number of interactions with the environment.
The Utility of Sparse Representations for Control in Reinforcement Learning
We investigate sparse representations for control in reinforcement learning.
Online Off-policy Prediction
The ability to learn behavior-contingent predictions online and off-policy has long been advocated as a key capability of predictive-knowledge learning systems but remained an open algorithmic challenge for decades.
Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement
We first provide a policy improvement result in an idealized setting, and then prove that our conditional CEM (CCEM) strategy tracks a CEM update per state, even with changing action-values.
Importance Resampling for Off-policy Policy Evaluation
We propose Importance Resampling (IR) for off-policy learning, that resamples experience from the replay buffer and applies a standard on-policy update.
High-confidence error estimates for learned value functions
We provide experiments investigating the number of samples required by this offline algorithm in simple benchmark reinforcement learning domains, and highlight that there are still many open questions to be solved for this important problem.
General Value Function Networks
A general purpose strategy for state construction is to learn the state update using a Recurrent Neural Network (RNN), which updates the internal state using the current internal state and the most recent observation.
Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control
Recent work has shown that reinforcement learning (RL) is a promising approach to control dynamical systems described by partial differential equations (PDE).
Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains
We show that a model, as opposed to a replay buffer, is particularly useful for specifying which states to sample from during planning, such as predecessor states that propagate information in reverse from a state more quickly.
Improving Regression Performance with Distributional Losses
We provide theoretical support for this alternative hypothesis, by characterizing the norm of the gradients of this loss.
Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods
This paper investigates estimating the variance of a temporal-difference learning agent's update target.
Discovery of Predictive Representations With a Network of General Value Functions
We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent's representation of the world.
Multi-view Matrix Factorization for Linear Dynamical System Estimation
In this paper, we instead reconsider likelihood maximization and develop an optimization based strategy for recovering the latent states and transition parameters.
Effective sketching methods for value function approximation
As a remedy, we demonstrate how to use sketching more sparingly, with only a left-sided sketch, that can still enable significant computational gains and the use of these matrix-based learning algorithms that are less sensitive to parameters.
Adapting Kernel Representations Online Using Submodular Maximization
In this work, we develop an approximately submodular criterion for this setting, and an efficient online greedy submodular maximization algorithm for optimizing the criterion.
Learning Sparse Representations in Reinforcement Learning with Sparse Coding
Outside of reinforcement learning, sparse coding representations have been widely used, with non-convex objectives that result in discriminative representations.
Recovering True Classifier Performance in Positive-Unlabeled Learning
A common approach in positive-unlabeled learning is to train a classification model between labeled and unlabeled data.
Accelerated Gradient Temporal Difference Learning
The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD({\lambda}) to data efficient least squares methods.
Unifying task specification in reinforcement learning
Reinforcement learning tasks are typically specified as Markov decision processes.
A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning
One of the main obstacles to broad application of reinforcement learning methods is the parameter sensitivity of our core learning algorithms.
Estimating the class prior and posterior from noisy positives and unlabeled data
We develop a classification algorithm for estimating posterior distributions from positive-unlabeled data, that is robust to noise in the positive labels and effective for high-dimensional data.
Identifying global optimality for dictionary learning
We then provide an empirical investigation into practical optimization choices for using alternating minimization for induced DLMs, for both batch and stochastic gradient descent.
Investigating practical linear temporal difference learning
First, we derive two new hybrid TD policy-evaluation algorithms, which fill a gap in this collection of algorithms.
Nonparametric semi-supervised learning of class proportions
This problem can be decomposed into two steps: (i) the development of accurate predictors that discriminate between positive and unlabeled data, and (ii) the accurate estimation of the prior probabilities of positive and negative examples.
Incremental Truncated LSTD
Balancing between computational efficiency and sample efficiency is an important goal in reinforcement learning.
Emphatic Temporal-Difference Learning
Emphatic algorithms are temporal-difference learning algorithms that change their effective state distribution by selectively emphasizing and de-emphasizing their updates on different time steps.
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning
In this paper we introduce the idea of improving the performance of parametric temporal-difference (TD) learning algorithms by selectively emphasizing or de-emphasizing their updates on different time steps.
Convex Multi-view Subspace Learning
Subspace learning seeks a low dimensional representation of data that enables accurate reconstruction.
Off-Policy Actor-Critic
Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning.
Interval Estimation for Reinforcement-Learning Algorithms in Continuous-State Domains
The reinforcement learning community has explored many approaches to obtain- ing value estimates and models to guide decision making; these approaches, how- ever, do not usually provide a measure of confidence in the estimate.
Relaxed Clipping: A Global Training Method for Robust Regression and Classification
We present a generic procedure that can be applied to standard loss functions and demonstrate improved robustness in regression and classification problems.
