Search Results for author:
Found 98 papers, 40 papers with code
Double Reinforcement Learning for Efficient and Robust Off-Policy Evaluation
Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible.
Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes
We characterize the sharp bounds on policy value under this model, that is, the tightest possible bounds given by the transition observations from the original MDP, and we study the estimation of these bounds from such transition observations.
Hessian-Free Laplace in Bayesian Deep Learning
The Laplace approximation (LA) of the Bayesian posterior is a Gaussian distribution centered at the maximum a posteriori estimate.
Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL
We study Risk-Sensitive Reinforcement Learning (RSRL) with the Optimized Certainty Equivalent (OCE) risk, which generalizes Conditional Value-at-risk (CVaR), entropic risk and Markowitz's mean-variance.
Switching the Loss Reduces the Cost in Batch Reinforcement Learning
We propose training fitted Q-iteration with log-loss (FQI-LOG) for batch reinforcement learning (RL).
Is Cosine-Similarity of Embeddings Really About Similarity?
Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations.
Applied Causal Inference Powered by ML and AI
An introduction to the emerging fusion of machine learning and causal inference.
More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning
Second-order bounds are instance-dependent bounds that scale with the variance of return, which we prove are tighter than the previously known small-loss bounds of distributional RL.
Distributional Reinforcement Learning
Multi-Armed Bandits
+1
Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams
We propose a novel nonparametric sequential test for composite hypotheses for means of multiple data streams.
Multi-Armed Bandits with Interference
Prior research on experimentation with interference has concentrated on the final output of a policy.
Clustered Switchback Experiments: Near-Optimal Rates Under Spatiotemporal Interference
We consider experimentation in the presence of non-stationarity, inter-unit (spatial) interference, and carry-over effects (temporal interference), where we wish to estimate the global average treatment effect (GATE), the difference between average outcomes having exposed all units at all times to treatment or to control.
Low-Rank MDPs with Continuous Action Spaces
Low-Rank Markov Decision Processes (MDPs) have recently emerged as a promising framework within the domain of reinforcement learning (RL), as they allow for provably approximately correct (PAC) learning guarantees while also incorporating ML algorithms for representation learning.
Off-Policy Evaluation for Large Action Spaces via Policy Convolution
To address these challenges, we introduce the Policy Convolution (PC) family of estimators.
Large Language Models as Zero-Shot Conversational Recommenders
In this paper, we present empirical studies on conversational recommendation tasks using representative large language models in a zero-shot setting with three primary contributions.
Source Condition Double Robust Inference on Functionals of Inverse Problems
We consider estimation of parameters defined as linear functionals of solutions to linear inverse problems.
JoinGym: An Efficient Query Optimization Environment for Reinforcement Learning
Join order selection (JOS) is the problem of ordering join operations to minimize total query execution cost and it is the core NP-hard combinatorial optimization problem of query optimization.
The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning
In online RL, we propose a DistRL algorithm that constructs confidence sets using maximum likelihood estimation.
Provable Offline Preference-Based Reinforcement Learning
Our proposed algorithm consists of two main steps: (1) estimate the implicit reward using Maximum Likelihood Estimation (MLE) with general function approximation from offline data and (2) solve a distributionally robust planning problem over a confidence set around the MLE.
B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Confounding
There has been recent progress on robust and efficient methods for estimating the conditional average treatment effect (CATE) function, but these methods often do not take into account the risk of hidden confounding, which could arbitrarily and unknowingly bias any causal estimate based on observational data.
Minimax Instrumental Variable Regression and $L_2$ Convergence Guarantees without Identification or Closedness
In this paper, we study nonparametric estimation of instrumental variable (IV) regressions.
Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR
In this paper, we study risk-sensitive Reinforcement Learning (RL), focusing on the objective of Conditional Value at Risk (CVaR) with risk tolerance $\tau$.
Smooth Non-Stationary Bandits
In many applications of online decision making, the environment is non-stationary and it is therefore crucial to use bandit algorithms that handle changes.
Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations
The type-I-error results primarily leverage a martingale strong invariance principle and establish that these tests (and their implied confidence sequences) have type-I error rates asymptotically equivalent to the desired (possibly varying) $\alpha$-level.
A Review of Off-Policy Evaluation in Reinforcement Learning
Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems.
The Implicit Delta Method
When the predictive model is simple and its evaluation differentiable, this task is solved by the delta method, where we propagate the asymptotically-normal uncertainty in the predictive model through the evaluation to compute standard errors and Wald confidence intervals.
Provable Safe Reinforcement Learning with Binary Feedback
Many existing approaches to safe RL rely on receiving numeric safety feedback, but in many cases this feedback can only take binary values; that is, whether an action in a given state is safe or unsafe.
Inference on Strongly Identified Functionals of Weakly Identified Functions
In a variety of applications, including nonparametric instrumental variable (NPIV) analysis, proximal causal inference under unmeasured confounding, and missing-not-at-random data with shadow variables, we are interested in inference on a continuous linear functional (e. g., average causal effects) of nuisance function (e. g., NPIV regression) defined by conditional moment restrictions.
Future-Dependent Value-Based Off-Policy Evaluation in POMDPs
Finally, we extend our methods to learning of dynamics and establish the connection between our approach and the well-known spectral learning methods in POMDPs.
Learning Bellman Complete Representations for Offline Policy Evaluation
We study representation learning for Offline Reinforcement Learning (RL), focusing on the important task of Offline Policy Evaluation (OPE).
Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
We show our algorithm's computational and statistical complexities scale polynomially with respect to the horizon and the intrinsic dimension of the feature on the observation space.
Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems
We study Reinforcement Learning for partially observable dynamical systems using function approximation.
Robust and Agnostic Learning of Conditional Distributional Treatment Effects
Our method is model-agnostic in that it can provide the best projection of CDTE onto the regression model class.
What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment
), whether exposed to the standard experience A or a new one B, hypothetically it could be because the change affects no one, because the change positively affects half the user population to go from no-click to click while negatively affecting the other half, or something in between.
Estimating Structural Disparities for Face Models
Performance of the model for each group is calculated by comparing $\hat{y}$ and $y$ for the datapoints within a specific group, and as a result, disparity of performance across the different groups can be calculated.
Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning
Thanks to a localization technique, LDR$^2$OPE only requires fitting a small number of regressions, just like DR methods for standard OPE.
Long-term Causal Inference Under Persistent Confounding via Data Combination
In this paper, we uniquely tackle the challenge of persistent unmeasured confounders, i. e., some unmeasured confounders that can simultaneously affect the treatment, short-term outcomes and the long-term outcome, noting that they invalidate identification strategies in previous literature.
Treatment Effect Risk: Bounds and Inference
Since the average treatment effect (ATE) measures the change in social welfare, even if positive, there is a risk of negative effect on, say, some 10% of the population.
Doubly-Valid/Doubly-Sharp Sensitivity Analysis for Causal Inference with Unmeasured Confounding
We consider the problem of constructing bounds on the average treatment effect (ATE) when unmeasured confounders exist but have bounded influence.
Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
To answer these, we extend the framework of proximal causal inference to our POMDP setting, providing a variety of settings where identification is made possible by the existence of so-called bridge functions.
Stateful Offline Contextual Policy Evaluation and Learning
We study off-policy evaluation and learning from sequential data in a structured class of Markov decision processes that arise from repeated interactions with an exogenous sequence of arrivals with contexts, which generate unknown individual-level responses to agent actions.
Residual Overfit Method of Exploration
The approach, which we term the residual overfit method of exploration (ROME), drives exploration towards actions where the overfit model exhibits the most overfitting compared to the tuned model.
Control Variates for Slate Off-Policy Evaluation
We study the problem of off-policy evaluation from batched contextual bandit data with multidimensional actions, often termed slates.
Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning
Empirical risk minimization (ERM) is the workhorse of machine learning, whether for classification and regression or for off-policy policy learning, but its model-agnostic guarantees can fail when we use adaptively collected data, such as the result of running a contextual bandit algorithm.
Post-Contextual-Bandit Inference
The adaptive nature of the data collected by contextual bandit algorithms, however, makes this difficult: standard estimators are no longer asymptotically normally distributed and classic confidence intervals fail to provide correct coverage.
Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach
Previous work has relied on completeness conditions on these functions to identify the causal parameters and required uniqueness assumptions in estimation, and they also focused on parametric estimation of bridge functions.
Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning using function approximation for marginal importance weights and $q$-functions when these are estimated using recent minimax methods.
Fast Rates for the Regret of Offline Reinforcement Learning
Second, we provide new analyses of FQI and Bellman residual minimization to establish the correct pointwise convergence guarantees.
Fairness, Welfare, and Equity in Personalized Pricing
These different application areas may lead to different concerns around fairness, welfare, and equity on different objectives: price burdens on consumers, price envy, firm revenue, access to a good, equal access, and distributional consequences when the good in question further impacts downstream outcomes of interest.
The Variational Method of Moments
The conditional moment problem is a powerful formulation for describing structural causal parameters in terms of observables, a prominent example being instrumental variable regression.
Rejoinder: New Objectives for Policy Learning
I provide a rejoinder for discussion of "More Efficient Policy Learning via Optimal Retargeting" to appear in the Journal of the American Statistical Association with discussion by Oliver Dukes and Stijn Vansteelandt; Sijia Li, Xiudi Li, and Alex Luedtkeand; and Muxuan Liang and Yingqi Zhao.
Fast Rates for Contextual Linear Optimization
While one may use off-the-shelf machine learning methods to separately learn a predictive model and plug it in, a variety of recent methods instead integrate estimation and optimization by fitting the model to directly optimize downstream decision performance.
Optimal Off-Policy Evaluation from Multiple Logging Policies
We study off-policy evaluation (OPE) from multiple logging policies, each generating a dataset of fixed size, i. e., stratified sampling.
Stochastic Optimization Forests
We study contextual stochastic optimization problems, where we leverage rich auxiliary observations (e. g., product characteristics) to improve decision making with uncertain variables (e. g., demand).
Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders.
Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
Targeting deterministic policies, for which action is a deterministic function of state, is crucial since optimal policies are always deterministic (up to ties).
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning
Compared with the classic case of a pre-specified evaluation policy, when evaluating natural stochastic policies, the efficiency bound, which measures the best-achievable estimation error, is inflated since the evaluation policy itself is unknown.
On the Optimality of Randomization in Experimental Design: How to Randomize for Minimax Variance and Design-Based Inference
When this set is permutation symmetric, the optimal design is complete randomization, and using a single partition (i. e., the design that only randomizes the treatment labels for each side of the partition) has minimax risk larger by a factor of $n-1$.
DTR Bandit: Learning to Make Response-Adaptive Decisions With Low Regret
While existing literature mostly focuses on estimating the optimal DTR from offline data such as from sequentially randomized trials, we study the problem of developing the optimal DTR in an online manner, where the interaction with each individual affect both our cumulative reward and our data collection for future learning.
On the role of surrogates in the efficient estimation of treatment effects with limited outcome data
However, there is often an abundance of observations on surrogate outcomes not of primary interest, such as short-term health effects or online-ad click-through.
Efficient Policy Learning from Surrogate-Loss Classification Reductions
We show that, under a correct specification assumption, the weighted classification formulation need not be efficient for policy parameters.
Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
We develop a robust approach that estimates sharp bounds on the (unidentifiable) value of a given policy in an infinite-horizon problem given data from another policy with unobserved confounding, subject to a sensitivity model.
Statistically Efficient Off-Policy Policy Gradients
Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value.
Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects
Practitioners in diverse fields such as healthcare, economics and education are eager to apply machine learning to improve decision making.
Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyond
A central example is the efficient estimating equation for the (local) quantile treatment effect ((L)QTE) in causal inference, which involves as a nuisance the covariate-conditional cumulative distribution function evaluated at the quantile to be estimated.
Assessing Disparate Impact of Personalized Interventions: Identifiability and Bounds
Personalized interventions in social services, education, and healthcare leverage individual-level causal effect predictions in order to give the best treatment to each individual or to prioritize program interventions for the individuals most likely to benefit.
Kernel Optimal Orthogonality Weighting: A Balancing Approach to Estimating Effects of Continuous Treatments
In this paper, we propose Kernel Optimal Orthogonality Weighting (KOOW), a convex optimization-based method, for estimating the effects of continuous treatments.
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
This precisely characterizes the curse of horizon: in time-variant processes, OPE is only feasible in the near-on-policy setting, where behavior and target policies are sufficiently similar.
Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimes
We study a nonparametric contextual bandit problem where the expected reward functions belong to a H\"older class with smoothness parameter $\beta$.
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible.
Optimal Estimation of Generalized Average Treatment Effects using Kernel Optimal Matching
In causal inference, a variety of causal effect estimands have been studied, including the sample, uncensored, target, conditional, optimal subpopulation, and optimal weighted average treatment effects.
Policy Evaluation with Latent Confounders via Optimal Balance
We study the question of policy evaluation when we instead have proxies for the latent confounders and develop an importance weighting method that avoids fitting a latent outcome regression model.
More Efficient Policy Learning via Optimal Retargeting
Policy learning can be used to extract individualized treatment regimes from observational data in healthcare, civics, e-commerce, and beyond.
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
We propose new estimators for OPE based on empirical likelihood that are always more efficient than IS, SNIS, and DR and satisfy the same stability and boundedness properties as SNIS.
Assessing Disparate Impacts of Personalized Interventions: Identifiability and Bounds
Personalized interventions in social services, education, and healthcare leverage individual-level causal effect predictions in order to give the best treatment to each individual or to prioritize program interventions for the individuals most likely to benefit.
Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination
In this paper we study a fundamental challenge to assessing disparate impacts in practice: protected class membership is often not observed in the data.
Data-Pooling in Stochastic Optimization
This intuition further suggests that data-pooling offers the most benefits when there are many problems, each of which has a small amount of relevant data.
Deep Generalized Method of Moments for Instrumental Variable Analysis
Instrumental variable analysis is a powerful tool for estimating causal effects when randomization or full control of confounders is not possible.
The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric
To better account for this, in this paper, we investigate the fairness of predictive risk scores from the point of view of a bipartite ranking task, where one seeks to rank positive examples higher than negative ones.
Classifying Treatment Responders Under Causal Effect Monotonicity
In the context of individual-level causal inference, we study the problem of predicting whether someone will respond or not to a treatment based on their features and past examples of features, treatment indicator (e. g., drug/no drug), and a binary outcome (e. g., recovery from disease).
Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved
We also propose an alternative weighted estimator that uses soft classification, and show that its bias arises simply from the conditional covariance of the outcome with the true class membership.
More robust estimation of sample average treatment effects using Kernel Optimal Matching in an observational study of spine surgical interventions
Inverse probability of treatment weighting (IPTW), which has been used to estimate sample average treatment effects (SATE) using observational data, tenuously relies on the positivity assumption and the correct specification of the treatment assignment model, both of which are problematic assumptions in many observational studies.
Methodology stat.ML, stat.ME, stat.AP
Removing Hidden Confounding by Experimental Grounding
We introduce a novel method of using limited experimental data to correct the hidden confounding in causal effect models trained on larger observational data, even if the observational data does not fully overlap with the experimental data.
Interval Estimation of Individual-Level Causal Effects Under Unobserved Confounding
We study the problem of learning conditional average treatment effects (CATE) from observational data with unobserved confounders.
Residual Unfairness in Fair Machine Learning from Prejudiced Data
We connect these lines of work and study the residual unfairness that arises when a fairness-adjusted predictor is not actually fair on the target population due to systematic censoring of training data by existing biased policies.
Optimal Balancing of Time-Dependent Confounders for Marginal Structural Models
Marginal structural models (MSMs) estimate the causal effect of a time-varying treatment in the presence of time-dependent confounding via weighted regression.
Causal Inference with Noisy and Missing Covariates via Matrix Factorization
Valid causal inference in observational studies often requires controlling for confounders.
Confounding-Robust Policy Improvement
We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding.
Learning Weighted Representations for Generalization Across Designs
We pose both of these problems as prediction under a shift in design.
Policy Evaluation and Optimization with Continuous Treatments
We study the problem of policy evaluation and learning from batched contextual bandit data when treatments are continuous, going beyond previous work on discrete treatments.
DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training
We study optimal covariate balance for causal inferences from observational data when rich covariates and complex relationships necessitate flexible modeling with neural networks.
Instrument-Armed Bandits
We argue for a particular kind of regret that captures the causal effect of treatments but show that standard MAB algorithms cannot achieve sublinear control on this regret.
Balanced Policy Evaluation and Learning
We propose a new, balance-based approach that too makes the data look like the new policy but does so directly by finding weights that optimize for balance between the weighted data and the target policy in the given, finite sample, which is equivalent to minimizing worst-case or posterior conditional mean square error.
Generalized Optimal Matching Methods for Causal Inference
We develop an encompassing framework for matching, covariate balancing, and doubly-robust methods for causal inference from observational data called generalized optimal matching (GOM).
Dynamic Assortment Personalization in High Dimensions
In the dynamic setting, we show that structure-aware dynamic assortment personalization can have regret that is an order of magnitude smaller than structure-ignorant approaches.
Recursive Partitioning for Personalization using Observational Data
We study the problem of learning to choose from m discrete treatment options (e. g., news item or medical drug) the one with best causal effect for a particular instance (e. g., user or patient) where the training data consists of passive observations of covariates, treatment, and the outcome of the treatment.
Revealed Preference at Scale: Learning Personalized Preferences from Assortment Choices
In our model, the preferences of each customer or segment follow a separate parametric choice model, but the underlying structure of these parameters over all the models has low dimension.
From Predictive to Prescriptive Analytics
To demonstrate the power of our approach in a real-world setting we study an inventory management problem faced by the distribution arm of an international media conglomerate, which ships an average of 1bil units per year.
