Search Results for author:
Found 41 papers, 13 papers with code
Double Reinforcement Learning for Efficient and Robust Off-Policy Evaluation
Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible.
Regularized DeepIV with Model Selection
In this paper, we study nonparametric estimation of instrumental variable (IV) regressions.
Feedback Efficient Online Fine-Tuning of Diffusion Models
It is natural to frame this as a reinforcement learning (RL) problem, in which the objective is to fine-tune a diffusion model to maximize a reward function that corresponds to some property.
Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control
Diffusion models excel at capturing complex data distributions, such as those of natural images and proteins.
Functional Graphical Models: Structure Enables Offline Data-Driven Optimization
This kind of data-driven optimization (DDO) presents a range of challenges beyond those in standard prediction problems, since we need models that successfully predict the performance of new designs that are better than the best designs seen in the training set.
Source Condition Double Robust Inference on Functionals of Inverse Problems
We consider estimation of parameters defined as linear functionals of solutions to linear inverse problems.
Off-Policy Evaluation of Ranking Policies under Diverse User Behavior
We show that the resulting estimator, which we call Adaptive IPS (AIPS), can be unbiased under any complex user behavior.
Provable Reward-Agnostic Preference-Based Reinforcement Learning
Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than explicit reward signals.
Provable Offline Preference-Based Reinforcement Learning
Our proposed algorithm consists of two main steps: (1) estimate the implicit reward using Maximum Likelihood Estimation (MLE) with general function approximation from offline data and (2) solve a distributionally robust planning problem over a confidence set around the MLE.
Distributional Offline Policy Evaluation with Predictive Error Guarantees
Our theoretical results show that for both finite-horizon and infinite-horizon discounted settings, FLE can learn distributions that are close to the ground truth under total variation distance and Wasserstein distance, respectively.
Minimax Instrumental Variable Regression and $L_2$ Convergence Guarantees without Identification or Closedness
In this paper, we study nonparametric estimation of instrumental variable (IV) regressions.
A Review of Off-Policy Evaluation in Reinforcement Learning
Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems.
Inference on Strongly Identified Functionals of Weakly Identified Functions
In a variety of applications, including nonparametric instrumental variable (NPIV) analysis, proximal causal inference under unmeasured confounding, and missing-not-at-random data with shadow variables, we are interested in inference on a continuous linear functional (e. g., average causal effects) of nuisance function (e. g., NPIV regression) defined by conditional moment restrictions.
Future-Dependent Value-Based Off-Policy Evaluation in POMDPs
Finally, we extend our methods to learning of dynamics and establish the connection between our approach and the well-known spectral learning methods in POMDPs.
PAC Reinforcement Learning for Predictive State Representations
We show that given a realizable model class, the sample complexity of learning the near optimal policy only scales polynomially with respect to the statistical complexity of the model class, without any explicit polynomial dependence on the size of the state and observation spaces.
Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems
We study Reinforcement Learning for partially observable dynamical systems using function approximation.
Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
We show our algorithm's computational and statistical complexities scale polynomially with respect to the horizon and the intrinsic dimension of the feature on the observation space.
Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach
We present BRIEE (Block-structured Representation learning with Interleaved Explore Exploit), an algorithm for efficient reinforcement learning in Markov Decision Processes with block-structured dynamics (i. e., Block MDPs), where rich observations are generated from a set of unknown latent states.
A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes
In this work, we first propose novel identification methods for OPE in POMDPs with latent confounders, by introducing bridge functions that link the target policy's value and the observed data distribution.
Representation Learning for Online and Offline RL in Low-rank MDPs
This work studies the question of Representation Learning in RL: how can we learn a compact low-dimensional representation such that on top of the representation we can perform RL procedures such as exploration and exploitation, in a sample efficient manner.
Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
Under the assumption that the ground truth model belongs to our function class (i. e., realizability in the function class), CPPO has a PAC guarantee with offline data only providing partial coverage, i. e., it can learn a policy that competes against any policy that is covered by the offline data.
Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy.
Mitigating Covariate Shift in Imitation Learning via Offline Data With Partial Coverage
Instead, the learner is presented with a static offline dataset of state-action-next state triples from a potentially less proficient behavior policy.
Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach
Previous work has relied on completeness conditions on these functions to identify the causal parameters and required uniqueness assumptions in estimation, and they also focused on parametric estimation of bridge functions.
Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning using function approximation for marginal importance weights and $q$-functions when these are estimated using recent minimax methods.
Fast Rates for the Regret of Offline Reinforcement Learning
Second, we provide new analyses of FQI and Bellman residual minimization to establish the correct pointwise convergence guarantees.
Optimal Off-Policy Evaluation from Multiple Logging Policies
We study off-policy evaluation (OPE) from multiple logging policies, each generating a dataset of fixed size, i. e., stratified sampling.
Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
Targeting deterministic policies, for which action is a deterministic function of state, is crucial since optimal policies are always deterministic (up to ties).
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning
Compared with the classic case of a pre-specified evaluation policy, when evaluating natural stochastic policies, the efficiency bound, which measures the best-achievable estimation error, is inflated since the evaluation policy itself is unknown.
Off-Policy Evaluation and Learning for External Validity under a Covariate Shift
Then, we propose doubly robust and efficient estimators for OPE and OPL under a covariate shift by using a nonparametric estimator of the density ratio between the historical and evaluation data distributions.
Statistically Efficient Off-Policy Policy Gradients
Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value.
Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyond
A central example is the efficient estimating equation for the (local) quantile treatment effect ((L)QTE) in causal inference, which involves as a nuisance the covariate-conditional cumulative distribution function evaluated at the quantile to be estimated.
Minimax Weight and Q-Function Learning for Off-Policy Evaluation
We provide theoretical investigations into off-policy evaluation in reinforcement learning using function approximators for (marginalized) importance weights and value functions.
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
This precisely characterizes the curse of horizon: in time-variant processes, OPE is only feasible in the near-on-policy setting, where behavior and target policies are sufficiently similar.
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible.
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
We propose new estimators for OPE based on empirical likelihood that are always more efficient than IS, SNIS, and DR and satisfy the same stability and boundedness properties as SNIS.
Information criteria for non-normalized models
However, model selection methods for general non-normalized models have not been proposed so far.
Imputation estimators for unnormalized models with missing data
We propose estimation methods for such unnormalized models with missing data.
Unified estimation framework for unnormalized models with statistical efficiency
The parameter estimation of unnormalized models is a challenging problem.
Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance
First, we propose a method for reducing asymptotic variance by estimating the parameters of the auxiliary distribution.
Generative Adversarial Nets from a Density Ratio Estimation Perspective
Generative adversarial networks (GANs) are successful deep generative models.
