Search Results for author:
Found 88 papers, 20 papers with code
Reinforcement Learning with Differential Privacy
Motivated by high-stakes decision-making domains like personalized medicine where user information is inherently sensitive, we design privacy preserving exploration policies for episodic reinforcement learning (RL).
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
Language model alignment methods, such as reinforcement learning from human feedback (RLHF), have led to impressive advances in language model capabilities, but existing techniques are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model plateaus or degrades over the course of the alignment process.
Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics
We study computationally and statistically efficient Reinforcement Learning algorithms for the linear Bellman Complete setting, a setting that uses linear function approximation to capture value functions and unifies existing models like linear Markov Decision Processes (MDP) and Linear Quadratic Regulators (LQR).
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
Reinforcement learning from human feedback (RLHF) has emerged as a central tool for language model alignment.
Rich-Observation Reinforcement Learning with Continuous Latent Dynamics
The core of our algorithm is a new representation learning objective; we show that prior representation learning schemes tailored to discrete dynamics do not naturally extend to the continuous setting.
Can large language models explore in-context?
We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making.
Scalable Online Exploration via Coverability
We propose exploration objectives -- policy optimization objectives that enable downstream maximization of any reward function -- as a conceptual framework to systematize the study of exploration.
Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learning
A pervasive phenomenon in machine learning applications is distribution shift, where training and deployment conditions for a machine learning model differ.
Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression
This work studies training instabilities of behavior cloning with deep neural networks.
Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits
We consider offline policy optimization (OPO) in contextual bandits, where one is given a fixed dataset of logged interactions.
Streaming Active Learning with Deep Neural Networks
Active learning is perhaps most naturally posed as an online learning problem.
Learning Hidden Markov Models Using Conditional Samples
In this paper, we depart from this setup and consider an interactive access model, in which the algorithm can query for samples from the conditional distributions of the HMMs.
Statistical Learning under Heterogeneous Distribution Shift
We show that, when the class $F$ is "simpler" than $G$ (measured, e. g., in terms of its metric entropy), our predictor is more resilient to heterogeneous covariate shifts} in which the shift in $\mathbf{x}$ is much greater than that in $\mathbf{y}$.
Transformers Learn Shortcuts to Automata
Algorithmic reasoning requires capabilities which are most naturally understood through recurrent models of computation, like the Turing machine.
Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction.
Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models
In many sequential decision-making tasks, the agent is not able to model the full complexity of the world, which consists of multitudes of relevant and irrelevant information.
On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RL
We study reward-free reinforcement learning (RL) under general non-linear function approximation, and establish sample efficiency and hardness results under various standard structural assumptions.
Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information
In real-world reinforcement learning applications the learner's observation space is ubiquitously high-dimensional with both relevant and irrelevant information about the task at hand.
A Complete Characterization of Linear Estimators for Offline Policy Evaluation
Offline policy evaluation is a fundamental statistical problem in reinforcement learning that involves estimating the value function of some decision-making policy given data collected by a potentially different policy.
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs.
Provable Reinforcement Learning with a Short-Term Memory
Real-world sequential decision making problems commonly involve partial observability, which requires the agent to maintain a memory of history in order to infer the latent states, plan and make good decisions.
Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizability
We study the $K$-armed contextual dueling bandit problem, a sequential decision making setting in which the learner uses contextual information to make two decisions, but only observes \emph{preference-based feedback} suggesting that one decision was better than the other.
Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
This led Chen and Jiang (2019) to conjecture that concentrability (the most standard notion of coverage) and realizability (the weakest representation condition) alone are not sufficient for sample-efficient offline RL.
Universal and data-adaptive algorithms for model selection in linear contextual bandits
In this paper, we introduce new algorithms that a) explore in a data-adaptive manner, and b) provide model selection guarantees of the form $\mathcal{O}(d^{\alpha} T^{1- \alpha})$ with no feature diversity conditions whatsoever, where $d$ denotes the dimension of the linear model and $T$ denotes the total number of rounds.
Anti-Concentrated Confidence Bonuses for Scalable Exploration
Intrinsic rewards play a central role in handling the exploration-exploitation trade-off when designing sequential decision-making algorithms, in both foundational theory and state-of-the-art deep reinforcement learning.
Provable RL with Exogenous Distractors via Multistep Inverse Dynamics
We initiate the formal study of latent state discovery in the presence of such exogenous noise sources by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich observation RL.
Sparsity in Partially Controllable Linear Systems
However, in practice, we often encounter systems in which a large set of state variables evolve exogenously and independently of the control inputs; such systems are only partially controllable.
Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics
We initiate the formal study of latent state discovery in the presence of such exogenous noise sources by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich observation RL.
Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination
A recurring theme in statistical learning, online learning, and beyond is that faster convergence rates are possible for problems with low noise, often quantified by the performance of the best hypothesis; such results are known as first-order or small-loss guarantees.
Bayesian decision-making under misspecified priors with applications to meta-learning
We prove that the expected reward accrued by Thompson sampling (TS) with a misspecified prior differs by at most $\tilde{\mathcal{O}}(H^2 \epsilon)$ from TS with a well specified prior, where $\epsilon$ is the total-variation distance between priors and $H$ is the learning horizon.
Investigating the Role of Negatives in Contrastive Representation Learning
We focus on disambiguating the role of one of these parameters: the number of negative examples.
Gone Fishing: Neural Active Learning with Fisher Embeddings
There is an increasing need for effective active learning algorithms that are compatible with deep neural networks.
Model-free Representation Learning and Exploration in Low-rank MDPs
In this work, we present the first model-free representation learning algorithms for low rank MDPs.
Learning the Linear Quadratic Regulator from Nonlinear Observations
We introduce a new algorithm, RichID, which learns a near-optimal policy for the RichLQR with sample complexity scaling only with the dimension of the latent state space and the capacity of the decoder function class.
Private Reinforcement Learning with PAC and Regret Guarantees
Motivated by high-stakes decision-making domains like personalized medicine where user information is inherently sensitive, we design privacy preserving exploration policies for episodic reinforcement learning (RL).
Contrastive learning, multi-view redundancy, and linear models
Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems.
Information Theoretic Regret Bounds for Online Nonlinear Control
This work studies the problem of sequential control in an unknown, nonlinear dynamical system, where we model the underlying system dynamics as an unknown function in a known Reproducing Kernel Hilbert Space.
Sample-Efficient Reinforcement Learning of Undercomplete POMDPs
Partial observability is a common challenge in many reinforcement learning applications, which requires an agent to maintain memory, infer latent states, and integrate this past information into exploration.
Open Problem: Model Selection for Contextual Bandits
In statistical learning, algorithms for model selection allow the learner to adapt to the complexity of the best hypothesis class in a sequence.
Provably adaptive reinforcement learning in metric spaces
We study reinforcement learning in continuous state and action spaces endowed with a metric.
FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
In order to deal with the curse of dimensionality in reinforcement learning (RL), it is common practice to make parametric assumptions where values or policies are functions of some low dimensional feature space.
Efficient Contextual Bandits with Continuous Actions
We create a computationally tractable algorithm for contextual bandits with continuous actions having unknown structure.
Contrastive estimation reveals topic posterior information to linear models
Contrastive learning is an approach to representation learning that utilizes naturally occurring similar and dissimilar pairs of data points to find useful embeddings of data.
Contextual Search in the Presence of Adversarial Corruptions
We initiate the study of contextual search when some of the agents can behave in ways inconsistent with the underlying response model.
Adaptive Estimator Selection for Off-Policy Evaluation
We develop a generic data-driven method for estimator selection in off-policy policy evaluation settings.
Reward-Free Exploration for Reinforcement Learning
We give an efficient algorithm that conducts $\tilde{\mathcal{O}}(S^2A\mathrm{poly}(H)/\epsilon^2)$ episodes of exploration and returns $\epsilon$-suboptimal policies for an arbitrary number of reward functions.
Algebraic and Analytic Approaches for Parameter Learning in Mixture Models
Our second approach uses algebraic and combinatorial tools and applies to binomial mixtures with shared trial parameter $N$ and differing success parameters, as well as to mixtures of geometric distributions.
Scalable Hierarchical Clustering with Tree Grafting
We introduce Grinch, a new algorithm for large-scale, non-greedy hierarchical clustering with general linkage functions that compute arbitrary similarity between two point sets.
Optimism in Reinforcement Learning with Generalized Linear Function Approximation
We design a new provably efficient algorithm for episodic reinforcement learning with generalized linear function approximation.
Sample Complexity of Learning Mixture of Sparse Linear Regressions
Ourtechniques are quite different from those in the previous work: for the noiselesscase, we rely on a property of sparse polynomials and for the noisy case, we providenew connections to learning Gaussian mixtures and use ideas from the theory of
Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning
We present an algorithm, HOMER, for exploration and reinforcement learning in rich observation environments that are summarizable by an unknown latent state space.
Sample Complexity of Learning Mixtures of Sparse Linear Regressions
In the problem of learning mixtures of linear regressions, the goal is to learn a collection of signal vectors from a sequence of (possibly noisy) linear measurements, where each measurement is evaluated on an unknown signal drawn uniformly from this collection.
Robust Dynamic Assortment Optimization in the Presence of Outlier Customers
We establish both upper and lower bounds on the regret, and show that our policy is optimal up to logarithmic factor in $T$ when the assortment capacity is constant.
Doubly robust off-policy evaluation with shrinkage
We propose a new framework for designing estimators for off-policy evaluation in contextual bandits.
Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
We design a new algorithm for batch active learning with deep neural network models.
Model selection for contextual bandits
We work in the stochastic realizable setting with a sequence of nested linear policy classes of dimension $d_1 < d_2 < \ldots$, where the $m^\star$-th class contains the optimal policy, and we design an algorithm that achieves $\tilde{O}(T^{2/3}d^{1/3}_{m^\star})$ regret with no prior knowledge of the optimal dimension $d_{m^\star}$.
Contextual Bandits with Continuous Actions: Smoothing, Zooming, and Adapting
We study contextual bandit learning with an abstract policy class and continuous action space.
Provably efficient RL with Rich Observations via Latent State Decoding
We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states.
Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
We study the sample complexity of model-based reinforcement learning (henceforth RL) in general contextual decision processes that require strategic exploration to find a near-optimal policy.
Contextual bandits with surrogate losses: Margin bounds and efficient algorithms
We use surrogate losses to obtain several new regret bounds and new algorithms for contextual bandit learning.
Myopic Bayesian Design of Experiments via Posterior Sampling and Probabilistic Programming
We design a new myopic strategy for a wide class of sequential design of experiment (DOE) problems, where the goal is to collect data in order to to fulfil a certain problem specific goal.
Semiparametric Contextual Bandits
This paper studies semiparametric contextual bandits, a generalization of the linear stochastic bandit problem where the reward for an action is modeled as a linear function of known action features confounded by an non-linear action-independent term.
On Oracle-Efficient PAC RL with Rich Observations
We study the computational tractability of PAC reinforcement learning with rich observations.
Disagreement-Based Combinatorial Pure Exploration: Sample Complexity Bounds and an Efficient Algorithm
We design new algorithms for the combinatorial pure exploration problem in the multi-arm bandit framework.
Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning
Knowledge bases (KB), both automatically and manually constructed, are often incomplete --- many valid facts can be inferred from the KB by synthesizing existing information.
Asynchronous Parallel Bayesian Optimisation via Thompson Sampling
We design and analyse variations of the classical Thompson sampling (TS) procedure for Bayesian optimisation (BO) in settings where function evaluations are expensive, but can be performed in parallel.
An Online Hierarchical Algorithm for Extreme Clustering
Many modern clustering methods scale well to a large number of data items, N, but not to a large number of clusters, K. This paper introduces PERCH, a new non-greedy algorithm for online hierarchical clustering that scales to both massive N and K--a problem setting we term extreme clustering.
Active Learning for Cost-Sensitive Classification
We design an active learning algorithm for cost-sensitive multiclass classification: problems where different errors have different costs.
Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
Our first contribution is a complexity measure, the Bellman rank, that we show enables tractable learning of near-optimal behavior in these processes and is naturally small for many well-studied reinforcement learning settings.
Improved Regret Bounds for Oracle-Based Adversarial Contextual Bandits
We give an oracle-based algorithm for the adversarial contextual bandit problem, where either contexts are drawn i. i. d.
Off-policy evaluation for slate recommendation
This paper studies the evaluation of policies that recommend an ordered set of items (e. g., a ranking) based on some context---a common scenario in web search, ads, and recommendation.
Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains
We address both of these challenges with two complementary techniques: First, we develop a gradient-boosting style, non-parametric function approximator for learning on $Q$-function residuals.
PAC Reinforcement Learning with Rich Observations
We prove that the algorithm learns near optimal behavior after a number of episodes that is polynomial in all relevant parameters, logarithmic in the number of policies, and independent of the size of the observation space.
Efficient Algorithms for Adversarial Contextual Learning
We provide the first oracle efficient sublinear regret algorithms for adversarial versions of the contextual bandit problem.
Nonparametric von Mises Estimators for Entropies, Divergences and Mutual Informations
We propose and analyse estimators for statistical functionals of one or moredistributions under nonparametric assumptions. Our estimators are derived from the von Mises expansion andare based on the theory of influence functions, which appearin the semiparametric statistics literature. We show that estimators based either on data-splitting or a leave-one-out techniqueenjoy fast rates of convergence and other favorable theoretical properties. We apply this framework to derive estimators for several popular informationtheoretic quantities, and via empirical evaluation, show the advantage of thisapproach over existing estimators.
Minimax Structured Normal Means Inference
We establish nearly matching upper and lower bounds on the minimax probability of error for any structured normal means problem, and we derive an optimality certificate for the maximum likelihood estimator, which can be applied to many instantiations.
Extreme Compressive Sampling for Covariance Estimation
This paper studies the problem of estimating the covariance of a collection of vectors using only highly compressed measurements of each vector.
Contextual Semibandits via Supervised Learning Oracles
We study an online decision making problem where on each round a learner chooses a list of items based on some side information, receives a scalar feedback value for each individual item, and a reward that is linearly related to this feedback.
Learning to Search Better Than Your Teacher
Methods for learning to search for structured prediction typically imitate a reference policy, with existing theoretical guarantees demonstrating low regret compared to that reference.
Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations
We propose and analyze estimators for statistical functionals of one or more distributions under nonparametric assumptions.
On Estimating $L_2^2$ Divergence
We give a comprehensive theoretical characterization of a nonparametric estimator for the $L_2^2$ divergence between two continuous distributions.
On the Power of Adaptivity in Matrix Completion and Approximation
We show that adaptive sampling allows one to eliminate standard incoherence assumptions on the matrix row space that are necessary for passive sampling procedures.
Subspace Learning from Extremely Compressed Measurements
Our theoretical results show that even a constant number of measurements per column suffices to approximate the principal subspace to arbitrary precision, provided that the number of vectors is large.
Nonparametric Estimation of Renyi Divergence and Friends
We consider nonparametric estimation of $L_2$, Renyi-$\alpha$ and Tsallis-$\alpha$ divergences between continuous distributions.
Near-optimal Anomaly Detection in Graphs using Lovasz Extended Scan Statistic
The detection of anomalous activity in graphs is a statistical problem that arises in many applications, such as network surveillance, disease outbreak detection, and activity monitoring in social networks.
Recovering Graph-Structured Activations using Adaptive Compressive Measurements
We study the localization of a cluster of activated vertices in a graph, from adaptively designed compressive measurements.
Low-Rank Matrix and Tensor Completion via Adaptive Sampling
In the absence of noise, we show that one can exactly recover a $n \times n$ matrix of rank $r$ from merely $\Omega(n r^{3/2}\log(r))$ matrix entries.
Noise Thresholds for Spectral Clustering
Although spectral clustering has enjoyed considerable empirical success in machine learning, its theoretical properties are not yet fully developed.
