Search Results for author:
Found 79 papers, 23 papers with code
Evaluating and Optimizing Educational Content with Large Language Model Judgments
We propose an alternative approach that uses Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes.
Experiment Planning with Function Approximation
We study the problem of experiment planning with function approximation in contextual bandit problems.
Adaptive Instrument Design for Indirect Experiments
Indirect experiments provide a valuable framework for estimating treatment effects in situations where conducting randomized control trials (RCTs) is impractical or unethical.
Adaptive Interventions with User-Defined Goals for Health Behavior Change
We prove that our modification incurs only a constant penalty on the cumulative regret while preserving the sample complexity benefits of data sharing.
Reinforcement Learning Tutor Better Supported Lower Performers in a Math Task
Resource limitations make it hard to provide all students with one of the most effective educational interventions: personalized instruction.
Estimating Optimal Policy Value in General Linear Contextual Bandits
Whether this is possible for more realistic context distributions has remained an open and important question for tasks such as model selection.
Model-based Offline Reinforcement Learning with Local Misspecification
We present a model-based offline reinforcement learning policy performance lower bound that explicitly captures dynamics model misspecification and distribution mismatch and we propose an empirical algorithm for optimal offline policy selection.
Giving Feedback on Interactive Student Programs with Meta-Exploration
However, teaching and giving feedback on such software is time-consuming -- standard approaches require instructors to manually grade student-implemented interactive programs.
Oracle Inequalities for Model Selection in Offline Reinforcement Learning
We propose the first model selection algorithm for offline RL that achieves minimax rate-optimal oracle inequalities up to logarithmic factors.
Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
Inspired by statistical model selection methods for supervised learning, we introduce a task- and method-agnostic pipeline for automatically training, comparing, selecting, and deploying the best policy when the provided dataset is limited in size.
Offline Policy Optimization with Eligible Actions
Offline policy optimization could have a large impact on many real-world decision-making problems, as online learning may be infeasible in many applications.
Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning
Online reinforcement learning (RL) algorithms are often difficult to deploy in complex human-facing applications as they may learn slowly and have poor early performance.
Reinforcement Learning with State Observation Costs in Action-Contingent Noiselessly Observable Markov Decision Processes
Many real-world problems that require making optimal sequences of decisions under uncertainty involve costs when the agent wishes to obtain information about its environment.
Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation
Off-policy policy evaluation methods for sequential decision making can be used to help identify if a proposed decision policy is better than a current baseline policy.
Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects
We propose rank-weighted average treatment effect (RATE) metrics as a simple and general family of metrics for comparing and testing the quality of treatment prioritization rules.
Play to Grade: Testing Coding Games as Classifying Markov Decision Process
Contemporary coding education often presents students with the task of developing programs that have user interaction and complex dynamic systems, such as mouse based games.
Avoiding Overfitting to the Importance Weights in Offline Policy Optimization
Offline policy optimization has a critical impact on many real-world decision-making problems, as online learning is costly and concerning in many applications.
Learning to be Fair: A Consequentialist Approach to Equitable Decision-Making
In our approach, one first elicits stakeholder preferences over the space of possible decisions and the resulting outcomes--such as preferences for balancing spending parity against court appearance rates.
Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Design of Experiments for Stochastic Contextual Linear Bandits
In the stochastic linear contextual bandit setting there exist several minimax procedures for exploration with policies that are reactive to the data being acquired.
Universal Off-Policy Evaluation
When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy.
Play to Grade: Grading Interactive Coding Games as Classifying Markov Decision Process
Contemporary coding education often present students with the task of developing programs that have user interaction and complex dynamic systems, such as mouse based games.
Provably Good Batch Off-Policy Reinforcement Learning Without Great Exploration
Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance.
Online Model Selection for Reinforcement Learning with Function Approximation
Towards this end, we consider the problem of model selection in RL with function approximation, given a set of candidate RL algorithms with known regret guarantees.
Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration
There has been growing progress on theoretical analyses for provably efficient learning in MDPs with linear function approximation, but much of the existing work has made strong assumptions to enable exploration by conventional exploration frameworks.
Provably Good Batch Reinforcement Learning Without Great Exploration
Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance.
Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
Model-based reinforcement learning (RL) is appealing because (i) it enables planning and thus more strategic exploration, and (ii) by decoupling dynamics from rewards, it enables fast transfer to new reward functions.
Power Constrained Bandits
However, when bandits are deployed in the context of a scientific study -- e. g. a clinical trial to test if a mobile health intervention is effective -- the aim is not only to personalize for an individual, but also to determine, with sufficient statistical power, whether or not the system's intervention is effective.
Value Driven Representation for Human-in-the-Loop Reinforcement Learning
In such systems there is typically an external human system designer that is creating, monitoring and modifying the interactive adaptive system, trying to improve its performance on the target outcomes.
Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
We assess robustness of OPE methods under unobserved confounding by developing worst-case bounds on the performance of an evaluation policy.
Learning Near Optimal Policies with Low Inherent Bellman Error
This has two important consequences: 1) it shows that exploration is possible using only \emph{batch assumptions} with an algorithm that achieves the optimal statistical rate for the setting we consider, which is more general than prior work on low-rank MDPs 2) the lack of closedness (measured by the inherent Bellman error) is only amplified by $\sqrt{d_t}$ despite working in the online setting.
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions
Off-policy evaluation in reinforcement learning offers the chance of using observational data to improve future outcomes in domains such as healthcare and education, but safe deployment in high stakes settings requires ways of assessing its validity.
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Accurate reporting of energy and carbon usage is essential for understanding the potential climate impacts of machine learning research.
Sublinear Optimal Policy Value Estimation in Contextual Bandits
We study the problem of estimating the expected reward of the optimal policy in the stochastic disjoint linear bandit setting.
Limiting Extrapolation in Linear Approximate Value Iteration
We prove that if the features at any state can be represented as a convex combination of features at the anchor points, then errors are propagated linearly over iterations (instead of exponentially) and our method achieves a polynomial sample complexity bound in the horizon and the number of anchor points.
Offline Contextual Bandits with High Probability Fairness Guarantees
We present RobinHood, an offline contextual bandit algorithm designed to satisfy a broad family of fairness constraints.
Almost Horizon-Free Structure-Aware Best Policy Identification with a Generative Model
This paper focuses on the problem of computing an $\epsilon$-optimal policy in a discounted Markov Decision Process (MDP) provided that we can access the reward and transition function through a generative model.
Missingness as Stability: Understanding the Structure of Missingness in Longitudinal EHR data and its Impact on Reinforcement Learning in Healthcare
There is an emerging trend in the reinforcement learning for healthcare literature.
Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy
While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications.
Problem Dependent Reinforcement Learning Bounds Which Can Identify Bandit Structure in MDPs
In order to make good decision under uncertainty an agent must learn from observations.
Frequentist Regret Bounds for Randomized Least-Squares Value Iteration
We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning (RL).
Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
Surprisingly, we find that in finite horizon MDPs there is no strict variance reduction of per-decision importance sampling or stationary importance sampling, comparing with vanilla importance sampling.
Directed Exploration for Reinforcement Learning
Efficient exploration is necessary to achieve good sample efficiency for reinforcement learning in general.
Learning When-to-Treat Policies
Many applied decision-making problems have a dynamic component: The policymaker needs not only to choose whom to treat, but also when to start which treatment.
Combining Parametric and Nonparametric Models for Off-Policy Evaluation
We consider a model-based approach to perform batch off-policy evaluation in reinforcement learning.
Learning Abstract Models for Long-Horizon Exploration
In our approach, a manager maintains an abstract MDP over a subset of the abstract states, which grows monotonically through targeted exploration (possible due to the abstract MDP).
Learning Procedural Abstractions and Evaluating Discrete Latent Temporal Structure
Given a dataset of time-series, the goal is to identify the latent sequence of steps common to them and label each time-series with the temporal extent of these procedural steps.
PLOTS: Procedure Learning from Observations using Subtask Structure
In many cases an intelligent agent may want to learn how to mimic a single observed demonstrated trajectory.
Off-Policy Policy Gradient with State Distribution Correction
We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method.
Separating value functions across time-scales
In settings where this bias is unacceptable - where the system must optimize for longer horizons at higher discounts - the target of the value function approximator may increase in variance leading to difficulties in learning.
Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds
Strong worst-case performance bounds for episodic reinforcement learning exist but fortunately in practice RL algorithms perform much better than such bounds would predict.
Distilling Information from a Flood: A Possibility for the Use of Meta-Analysis and Systematic Review in Machine Learning Research
The current flood of information in all areas of machine learning research, from computer vision to reinforcement learning, has made it difficult to make aggregate scientific inferences.
Policy Certificates: Towards Accountable Reinforcement Learning
The performance of a reinforcement learning algorithm can vary drastically during learning because of exploration.
Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
In this work, we consider the problem of estimating a behaviour policy for use in Off-Policy Policy Evaluation (OPE) when the true behaviour policy is unknown.
Decoupling Gradient-Like Learning Rules from Representations
When creating a machine learning system, we must make two decisions: what representation should be used (i. e., what parameterized function should be used) and what learning rule should be used to search through the resulting set of representable functions.
Fast Exploration with Simplified Models and Approximately Optimistic Planning in Model Based Reinforcement Learning
People seem to build simple models that are easy to learn to support planning and strategic exploration.
Representation Balancing MDPs for Off-Policy Policy Evaluation
We study the problem of off-policy policy evaluation (OPPE) in RL.
When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithms
Efficient exploration is one of the key challenges for reinforcement learning (RL) algorithms.
Regret Minimization in MDPs with Options without Prior Knowledge
The option framework integrates temporal abstraction into the reinforcement learning model through the introduction of macro-actions (i. e., options).
Generalized Grounding Graphs: A Probabilistic Framework for Understanding Grounded Commands
Symbolic models capture linguistic structure but have not scaled successfully to handle the diverse language produced by untrained users.
On Ensuring that Intelligent Machines Are Well-Behaved
We propose a new framework for designing machine learning algorithms that simplifies the problem of specifying and regulating undesirable behaviors.
Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselines
We show how an action-dependent baseline can be used by the policy gradient theorem using function approximation, originally presented with action-independent baselines by (Sutton et al. 2000).
Decoupling Learning Rules from Representations
When creating an artificial intelligence system, we must make two decisions: what representation should be used (i. e., what parameterized function should be used) and what learning rule should be used to search through the resulting set of representable functions.
Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
Statistical performance bounds for reinforcement learning (RL) algorithms can be critical for high-stakes applications like healthcare.
Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
In addition, we can take advantage of special cases that arise due to options-based policies to further improve the performance of importance sampling.
Sample Efficient Feature Selection for Factored MDPs
This can result in a much better sample complexity when the in-degree of the necessary features is smaller than the in-degree of all features.
Sample Efficient Policy Search for Optimal Stopping Domains
Optimal stopping problems consider the question of deciding when to stop an observation-generating process in order to maximize a return.
Importance Sampling with Unequal Support
Importance sampling is often used in machine learning when training and testing data come from different distributions.
A PAC RL Algorithm for Episodic POMDPs
Many interesting real world domains involve reinforcement learning (RL) in partially observable environments.
Latent Contextual Bandits and their Application to Personalized Recommendations for New Users
We consider both the benefit of leveraging a set of learned latent user classes for new users, and how we can learn such latent classes from prior users.
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
In this paper we present a new way of predicting the performance of a reinforcement learning policy given historical data that may have been generated by a different policy.
Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
In this paper, we derive an upper PAC bound $\tilde O(\frac{|\mathcal S|^2 |\mathcal A| H^2}{\epsilon^2} \ln\frac 1 \delta)$ and a lower PAC bound $\tilde \Omega(\frac{|\mathcal S| |\mathcal A| H^2}{\epsilon^2} \ln \frac 1 {\delta + c})$ that match up to log-terms and an additional linear dependency on the number of states $|\mathcal S|$.
The Online Coupon-Collector Problem and Its Application to Lifelong Reinforcement Learning
Transferring knowledge across a sequence of related tasks is an important challenge in reinforcement learning (RL).
Online Stochastic Optimization under Correlated Bandit Feedback
In this paper we consider the problem of online stochastic optimization of a locally smooth function under bandit feedback.
Efficient Planning under Uncertainty with Macro-actions
We also demonstrate our algorithm being used to control a real robotic helicopter in a target monitoring experiment, which suggests that our approach has practical potential for planning in real-world, large partially observable domains where a multi-step lookahead is required to achieve good performance.
Sample Complexity of Multi-task Reinforcement Learning
Transferring knowledge across a sequence of reinforcement-learning tasks is challenging, and has a number of important applications.
Sequential Transfer in Multi-armed Bandit with Finite Set of Models
Learning from prior tasks and transferring that experience to improve future performance is critical for building lifelong learning agents.
Regret Bounds for Reinforcement Learning with Policy Advice
In some reinforcement learning problems an agent may be provided with a set of input policies, perhaps learned from prior experience or provided by advisors.
