Search Results for author:
Found 82 papers, 12 papers with code
The Need for a Big World Simulator: A Scientific Challenge for Continual Learning
The "small agent, big world" frame offers a conceptual view that motivates the need for continual learning.
Information-Theoretic Foundations for Machine Learning
Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning.
Exploration Unbound
A sequential decision-making agent balances between exploring to gain new knowledge about an environment and exploiting current knowledge to maximize immediate reward.
Satisficing Exploration for Deep Reinforcement Learning
In this work, we remedy this issue by extending an agent that directly represents uncertainty over the optimal value function allowing it to both bypass the need for model-based planning and to learn satisficing policies.
Efficient Exploration for LLMs
We present evidence of substantial benefit from efficient exploration in gathering human feedback to improve large language models.
An Information-Theoretic Analysis of In-Context Learning
Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted.
Adaptive Crowdsourcing Via Self-Supervised Learning
Common crowdsourcing systems average estimates of a latent quantity of interest provided by many crowdworkers to produce a group estimate.
RLHF and IIA: Perverse Incentives
Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA).
Non-Stationary Contextual Bandit Learning via Neural Predictive Ensemble Sampling
Real-world applications of contextual bandits often exhibit non-stationarity due to seasonality, serendipity, and evolving social trends.
Maintaining Plasticity in Continual Learning via Regenerative Regularization
In this paper, we propose L2 Init, a simple approach for maintaining plasticity by incorporating in the loss function L2 regularization toward initial parameters.
A Definition of Continual Reinforcement Learning
Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents.
On the Convergence of Bounded Agents
Standard models of the reinforcement learning problem give rise to a straightforward definition of convergence: An agent converges when its behavior or performance in each environment state stops changing.
Continual Learning as Computationally Constrained Reinforcement Learning
The design of such agents, which remains a long-standing challenge of artificial intelligence, is addressed by the subject of continual learning.
Scalable Neural Contextual Bandit for Recommender Systems
In two distinct large-scale experiments with real-world tasks, ENR significantly boosts click-through rates and user ratings by at least 9% and 6% respectively compared to state-of-the-art neural contextual bandit algorithms.
Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
In this work, we adopt a novel perspective wherein a pre-trained language model is itself simultaneously a policy, reward function, and transition function.
Bayesian Reinforcement Learning with Limited Cognitive Load
All biological and artificial agents must learn and make decisions given limits on their ability to process information.
A Definition of Non-Stationary Bandits
Despite the subject of non-stationary bandit learning having attracted much recent attention, we have yet to identify a formal definition of non-stationarity that can consistently distinguish non-stationary bandits from stationary ones.
Approximate Thompson Sampling via Epistemic Neural Networks
Further, we demonstrate that the \textit{epinet} -- a small additive network that estimates uncertainty -- matches the performance of large ensembles at orders of magnitude lower computational cost.
Leveraging Demonstrations to Improve Online Learning: Quality Matters
This offers insight into how pretraining can greatly improve online performance and how the degree of improvement increases with the expert's competence level.
Inclusive Artificial Intelligence
Prevailing methods for assessing and comparing generative AIs incentivize responses that serve a hypothetical representative individual.
An Information-Theoretic Analysis of Compute-Optimal Neural Scaling Laws
For a particular learning model inspired by barron 1993, we establish an upper bound on the minimal information-theoretically achievable expected error as a function of model and data set sizes.
Posterior Sampling for Continuing Environments
We develop an extension of posterior sampling for reinforcement learning (PSRL) that is suited for a continuing agent-environment interface and integrates naturally into agent designs that scale to complex environments.
Fine-Tuning Language Models via Epistemic Neural Networks
Language models often pre-train on large unsupervised text corpora, then fine-tune on additional task-specific data.
On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learning
Throughout the cognitive-science literature, there is widespread agreement that decision-making agents operating in the real world do so under limited information-processing capabilities and without access to unbounded cognitive or computational resources.
Is Stochastic Gradient Descent Near Optimal?
However, existing computational theory suggests that, even for single-hidden-layer teacher networks, to attain small error for all such teacher networks, the computation required to achieve this sample complexity is intractable.
Robustness of Epinets against Distributional Shifts
However, these improvements are relatively small compared to the outstanding issues in distributionally-robust deep learning.
Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping
In machine learning, an agent needs to estimate uncertainty to efficiently explore and adapt and to make effective decisions.
Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learning
To address this problem, we introduce an algorithm that, using rate-distortion theory, iteratively computes an approximately-value-equivalent, lossy compression of the environment which an agent may feasibly target in lieu of the true model.
Between Rate-Distortion Theory & Value Equivalence in Model-Based Reinforcement Learning
The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment.
Non-Stationary Bandit Learning via Predictive Sampling
We attribute such failures to the fact that, when exploring, the algorithm does not differentiate actions based on how quickly the information acquired loses its usefulness due to non-stationarity.
An Analysis of Ensemble Sampling
Ensemble sampling serves as a practical approximation to Thompson sampling when maintaining an exact posterior distribution over model parameters is computationally intractable.
An Information-Theoretic Framework for Supervised Learning
For a particular prior distribution on weights, we establish sample complexity bounds that are simultaneously width independent and linear in depth.
Evaluating High-Order Predictive Distributions in Deep Learning
Previous work has developed methods for assessing low-order predictive distributions with inputs sampled i. i. d.
Gaussian Imagination in Bandit Learning
We study the performance of an agent that attains a bounded information ratio with respect to a bandit environment with a Gaussian prior distribution and a Gaussian likelihood function when applied instead to a Bernoulli bandit.
The Value of Information When Deciding What to Learn
All sequential decision-making agents explore so as to acquire knowledge about a particular target.
The Neural Testbed: Evaluating Joint Predictions
Predictive distributions quantify uncertainties ignored by point estimates.
Evaluating Predictive Distributions: Does Bayesian Deep Learning Work?
This paper introduces \textit{The Neural Testbed}, which provides tools for the systematic evaluation of agents that generate such predictions.
Deep Exploration for Recommendation Systems
Where past work has aimed to learn from subsequent behavior, there has been a lack of effective methods for probing to elicit informative delayed feedback.
From Predictions to Decisions: The Importance of Joint Predictive Distributions
A fundamental challenge for any intelligent system is prediction: given some inputs, can you predict corresponding outcomes?
Epistemic Neural Networks
We introduce the epinet: an architecture that can supplement any conventional neural network, including large pretrained models, and can be trained with modest incremental computation to estimate uncertainty.
Reinforcement Learning, Bit by Bit
To illustrate concepts, we design simple agents that build on them and present computational results that highlight data efficiency.
A Bit Better? Quantifying Information for Bandit Learning
The information ratio offers an approach to assessing the efficacy with which an agent balances between exploration and exploitation.
Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent States
The time it takes to approach asymptotic performance is polynomial in the complexity of the agent's state representation and the time required to evaluate the best policy that the agent can represent.
Deciding What to Learn: A Rate-Distortion Approach
Agents that learn to select optimal actions represent a prominent focus of the sequential decision-making literature.
On Efficiency in Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) approaches promise to provide more efficient solutions to sequential decision making problems, both in terms of statistical as well as computational efficiency.
Randomized Value Functions via Posterior State-Abstraction Sampling
State abstraction has been an essential tool for dramatically improving the sample efficiency of reinforcement-learning algorithms.
Hypermodels for Exploration
This generalizes and extends the use of ensembles to approximate Thompson sampling.
Langevin DQN
Algorithms that tackle deep exploration -- an important challenge in reinforcement learning -- have relied on epistemic uncertainty representation through ensembles or other hypermodels, exploration bonuses, or visitation count distributions.
Provably Efficient Reinforcement Learning with Aggregated States
We establish that an optimistic variant of Q-learning applied to a fixed-horizon episodic Markov decision process with an aggregated state representation incurs regret $\tilde{\mathcal{O}}(\sqrt{H^5 M K} + \epsilon HK)$, where $H$ is the horizon, $M$ is the number of aggregate states, $K$ is the number of episodes, and $\epsilon$ is the largest difference between any pair of optimal state-action values associated with a common aggregate state.
Information-Theoretic Confidence Bounds for Reinforcement Learning
We integrate information-theoretic concepts into the design and analysis of optimistic algorithms and Thompson sampling.
Comments on the Du-Kakade-Wang-Yang Lower Bounds
Du, Kakade, Wang, and Yang recently established intriguing lower bounds on sample complexity, which suggest that reinforcement learning with a misspecified representation is intractable.
Behaviour Suite for Reinforcement Learning
bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives.
On the Performance of Thompson Sampling on Logistic Bandits
Specifically, we establish that, when the set of feasible actions is identical to the set of possible coefficient vectors, the Bayesian regret of Thompson sampling is $\tilde{O}(d\sqrt{T})$.
An Information-Theoretic Analysis for Thompson Sampling with Many Actions
We also offer a bound for the logistic bandit that dramatically improves on the best previously available, though this bound depends on an information-theoretic statistic that we have only been able to quantify via computation.
Scalable Coordinated Exploration in Concurrent Reinforcement Learning
We consider a team of reinforcement learning agents that concurrently operate in a common environment, and we develop an approach to efficient coordinated exploration that is suitable for problems of practical scale.
Satisficing in Time-Sensitive Bandit Learning
Much of the recent literature on bandit learning focuses on algorithms that aim to converge on an optimal action.
Coordinated Exploration in Concurrent Reinforcement Learning
We consider a team of reinforcement learning agents that concurrently learn to operate in a common environment.
Learning to Price with Reference Effects
As a firm varies the price of a product, consumers exhibit reference effects, making purchase decisions based not only on the prevailing price but also the product's price history.
A Tutorial on Thompson Sampling
Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance.
On Optimistic versus Randomized Exploration in Reinforcement Learning
We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning.
Ensemble Sampling
Thompson sampling has emerged as an effective heuristic for a broad range of online decision problems.
Time-Sensitive Bandit Learning and Satisficing Thompson Sampling
We propose satisficing Thompson sampling -- a variation of Thompson sampling -- and establish a strong discounted regret bound for this new algorithm.
Deep Exploration via Randomized Value Functions
We study the use of randomized value functions to guide deep exploration in reinforcement learning.
Gaussian-Dirichlet Posterior Dominance in Sequential Learning
We consider the problem of sequential learning from categorical observations bounded in [0, 1].
Conservative Contextual Linear Bandits
We prove an upper-bound on the regret of CLUCB and show that it can be decomposed into two terms: 1) an upper-bound for the regret of the standard linear UCB algorithm that grows with the time horizon and 2) a constant (does not grow with the time horizon) term that accounts for the loss of being conservative in order to satisfy the safety constraint.
On Lower Bounds for Regret in Reinforcement Learning
This is a brief technical note to clarify the state of lower bounds on regret for reinforcement learning.
Posterior Sampling for Reinforcement Learning Without Episodes
- Review similar results for optimistic algorithms in infinite horizon problems (Jaksch et al 2010, Bartlett and Tewari 2009, Abbasi-Yadkori and Szepesvari 2011), with particular attention to the dynamic episode growth.
Why is Posterior Sampling Better than Optimism for Reinforcement Learning?
Computational results demonstrate that posterior sampling for reinforcement learning (PSRL) dramatically outperforms algorithms driven by optimism, such as UCRL2.
Deep Exploration via Bootstrapped DQN
Efficient exploration in complex environments remains a major challenge for reinforcement learning.
Ranked #5 on
Atari Games
on Atari 2600 Breakout
Bootstrapped Thompson Sampling and Deep Exploration
This technical note presents a new approach to carrying out the kind of exploration achieved by Thompson sampling, but without explicitly maintaining or sampling from posterior distributions.
Model-based Reinforcement Learning and the Eluder Dimension
We consider the problem of learning to optimize an unknown Markov decision process (MDP).
Model-based Reinforcement Learning
reinforcement-learning
+1
An Information-Theoretic Analysis of Thompson Sampling
We provide an information-theoretic analysis of Thompson sampling that applies across a broad range of online optimization problems in which a decision-maker must learn from partial feedback.
Learning to Optimize via Information-Directed Sampling
We propose information-directed sampling -- a new approach to online optimization problems in which a decision-maker must balance between exploration and exploitation while learning from partial feedback.
Near-optimal Reinforcement Learning in Factored MDPs
Any reinforcement learning algorithm that applies to all Markov decision processes (MDPs) will suffer $\Omega(\sqrt{SAT})$ regret on some MDP, where $T$ is the elapsed time and $S$ and $A$ are the cardinalities of the state and action spaces.
Generalization and Exploration via Randomized Value Functions
We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions.
Efficient Exploration and Value Function Generalization in Deterministic Systems
We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization.
Eluder Dimension and the Sample Complexity of Optimistic Exploration
This paper considers the sample complexity of the multi-armed bandit with dependencies among the arms.
Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization
We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization.
(More) Efficient Reinforcement Learning via Posterior Sampling
This bound is one of the first for an algorithm not based on optimism, and close to the state of the art for any reinforcement learning algorithm.
Efficient Reinforcement Learning for High Dimensional Linear Quadratic Systems
In particular, our algorithm has an average cost of $(1+\eps)$ times the optimum cost after $T = \polylog(p) O(1/\eps^2)$.
Learning to Optimize Via Posterior Sampling
This paper considers the use of a simple posterior sampling algorithm to balance between exploration and exploitation when learning to optimize actions such as in multi-armed bandit problems.
Adaptive Execution: Exploration and Learning of Price Impact
The trader must learn coefficients of a price impact model while trading.
Trading and Market Microstructure
