Search Results for author:
Found 93 papers, 24 papers with code
Ordering-based Conditions for Global Convergence of Policy Gradient Methods
We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the policy update and the representation.
Mitigating Preference Hacking in Policy Optimization with Pessimism
This work tackles the problem of overoptimization in reinforcement learning from human feedback (RLHF), a prevalent technique for aligning models with human preferences.
Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE).
Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates
The proofs are based on novel findings about action sampling rates and the relationship between cumulative progress and noise, and extend the current understanding of how simple stochastic gradient methods behave in bandit settings.
Design Considerations in Offline Preference-based RL
Offline algorithms for Reinforcement Learning from Human Preferences (RLHF), which use only a fixed dataset of sampled responses given an input, and preference feedback among these responses, have gained increasing prominence in the literature on aligning language models.
Catoni Contextual Bandits are Robust to Heavy-tailed Rewards
When the variance of the reward at each round is known, we use a variance-weighted regression approach and establish a regret bound that depends only on the cumulative reward variance and logarithmically on the reward range $R$ as well as the number of rounds $T$.
Preserving Expert-Level Privacy in Offline Reinforcement Learning
The offline reinforcement learning (RL) problem aims to learn an optimal policy from historical data collected by one or more behavioural policies (experts) by interacting with an environment.
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
Our key insight is that, to be effective, the process reward for a step should measure progress: a change in the likelihood of producing a correct response in the future, before and after taking the step, corresponding to the notion of step-level advantages in RL.
Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
Reward-based finetuning is crucial for aligning language policies with intended behaviors (e. g., creativity and safety).
Robust Preference Optimization through Reward Model Distillation
Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models that, as a whole, is likely to include at least one reasonable proxy for the preference distribution.
Offline Imitation Learning from Multiple Baselines with Applications to Compiler Optimization
This work studies a Reinforcement Learning (RL) problem in which we are given a set of trajectories collected with K baseline policies.
Stochastic Gradient Succeeds for Bandits
We show that the \emph{stochastic gradient} bandit algorithm converges to a \emph{globally optimal} policy at an $O(1/t)$ rate, even with a \emph{constant} step size.
More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning
Second-order bounds are instance-dependent bounds that scale with the variance of return, which we prove are tighter than the previously known small-loss bounds of distributional RL.
Distributional Reinforcement Learning
Multi-Armed Bandits
+1
A Minimaximalist Approach to Reinforcement Learning from Human Feedback
Our approach is maximalist in that it provably handles non-Markovian, intransitive, and stochastic preferences while being robust to the compounding errors that plague offline approaches to sequential prediction.
Theoretical guarantees on the best-of-n alignment policy
A simple and effective method for the inference-time alignment of generative models is the best-of-$n$ policy, where $n$ samples are drawn from a reference policy, ranked based on a reward function, and the highest ranking one is selected.
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.
Efficient End-to-End Visual Document Understanding with Rationale Distillation
Pre-processing tools, such as optical character recognition (OCR), can map document image inputs to textual tokens, then large language models (LLMs) can reason over text.
A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks
We study the phenomenon of \textit{in-context learning} (ICL) exhibited by large language models, where they can adapt to a new learning task, given a handful of labeled examples, without any explicit parameter optimization.
An Empirical Evaluation of Federated Contextual Bandit Algorithms
As the adoption of federated learning increases for learning from sensitive data local to user devices, it is natural to ask if the learning can be done using implicit signals generated as users interact with the applications of interest, rather than requiring access to explicit labels which can be difficult to acquire in many tasks.
Leveraging User-Triggered Supervision in Contextual Bandits
We study contextual bandit (CB) problems, where the user can sometimes respond with the best action in a given context.
Learning in POMDPs is Sample-Efficient with Hindsight Observability
POMDPs capture a broad class of decision making problems, but hardness results suggest that learning is intractable even in simple settings due to the inherent partial observability.
VO$Q$L: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation
We study time-inhomogeneous episodic reinforcement learning (RL) under general function approximation and sparse rewards.
On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RL
We study reward-free reinforcement learning (RL) under general non-linear function approximation, and establish sample efficiency and hardness results under various standard structural assumptions.
Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexity
We propose a general framework to design posterior sampling methods for model-based RL.
Provable Benefits of Representational Transfer in Reinforcement Learning
We study the problem of representational transfer in RL, where an agent first pretrains in a number of source tasks to discover a shared representation, which is subsequently used to learn a good policy in a \emph{target task}.
Non-Linear Reinforcement Learning in Large Action Spaces: Structural Conditions and Sample-efficiency of Posterior Sampling
Provably sample-efficient Reinforcement Learning (RL) with rich observations and function approximation has witnessed tremendous recent progress, particularly when the underlying function approximators are linear.
Minimax Regret Optimization for Robust Machine Learning under Distribution Shift
We instead propose an alternative method called Minimax Regret Optimization (MRO), and show that under suitable conditions this method achieves uniformly low regret across all test distributions.
Adversarially Trained Actor Critic for Offline Reinforcement Learning
We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism.
Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach
We present BRIEE (Block-structured Representation learning with Interleaved Explore Exploit), an algorithm for efficient reinforcement learning in Markov Decision Processes with block-structured dynamics (i. e., Block MDPs), where rich observations are generated from a set of unknown latent states.
Provable RL with Exogenous Distractors via Multistep Inverse Dynamics
We initiate the formal study of latent state discovery in the presence of such exogenous noise sources by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich observation RL.
Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics
We initiate the formal study of latent state discovery in the presence of such exogenous noise sources by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich observation RL.
Bellman-consistent Pessimism for Offline Reinforcement Learning
The use of pessimism, when reasoning about datasets lacking exhaustive exploration has recently gained prominence in offline reinforcement learning.
Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximation
Policy optimization methods are popular reinforcement learning algorithms, because their incremental and on-policy nature makes them more stable than the value-based counterparts.
Provably Correct Optimization and Exploration with Non-linear Policies
Policy optimization methods remain a powerful workhorse in empirical Reinforcement Learning (RL), with a focus on neural policies that can easily reason over complex and continuous state and/or action spaces.
Towards a Dimension-Free Understanding of Adaptive Linear Control
We study the problem of adaptive control of the linear quadratic regulator for systems in very high, or even infinite dimension.
Model-free Representation Learning and Exploration in Low-rank MDPs
In this work, we present the first model-free representation learning algorithms for low rank MDPs.
Provably Good Batch Off-Policy Reinforcement Learning Without Great Exploration
Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance.
Provably Good Batch Reinforcement Learning Without Great Exploration
Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance.
PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
Direct policy gradient methods for reinforcement learning are a successful approach for a variety of reasons: they are model free, they directly optimize the performance metric of interest, and they allow for richly parameterized policies.
Policy Improvement via Imitation of Multiple Oracles
In this paper, we propose the state-wise maximum of the oracle policies' values as a natural baseline to resolve conflicting advice from multiple oracles.
Safe Reinforcement Learning via Curriculum Induction
In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly.
Optimizing Interactive Systems via Data-Driven Objectives
Effective optimization is essential for real-world interactive systems to provide a satisfactory user experience in response to changing user behavior.
Reparameterized Variational Divergence Minimization for Stable Imitation
While recent state-of-the-art results for adversarial imitation-learning algorithms are encouraging, recent works exploring the imitation learning from observation (ILO) setting, where trajectories \textit{only} contain expert observations, have not been met with the same success.
FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
In order to deal with the curse of dimensionality in reinforcement learning (RL), it is common practice to make parametric assumptions where values or policies are functions of some low dimensional feature space.
Federated Residual Learning
We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model.
Taking a hint: How to leverage loss predictors in contextual bandits?
We initiate the study of learning in contextual bandits with the help of loss predictors.
On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
Policy gradient methods are among the most effective methods in challenging reinforcement learning problems with large state and/or action spaces.
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
A standard technique to correct this bias is importance sampling, where samples from the model are weighted by the likelihood ratio under model and true distributions.
Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal
In this work, we study the effectiveness of the most natural plug-in approach to model-based planning: we build the maximum likelihood estimate of the transition model in the MDP from observations and then find an optimal policy in this empirical MDP.
Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
We design a new algorithm for batch active learning with deep neural network models.
Fair Regression: Quantitative Definitions and Reduction-based Algorithms
Our schemes only require access to standard risk minimization algorithms (such as standard classification or least-squares regression) while providing theoretical guarantees on the optimality and fairness of the obtained solutions.
Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations
We address the opportunity to maximize the utility of an overall computing system by employing reinforcement learning to guide the configuration of the set of interacting modules that comprise the system.
Off-Policy Policy Gradient with State Distribution Correction
We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method.
Bias Correction of Learned Generative Models via Likelihood-free Importance Weighting
A standard technique to correct this bias is by importance weighting samples from the model by the likelihood ratio under the model and true distributions.
Provably efficient RL with Rich Observations via Latent State Decoding
We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states.
Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedback
We investigate the feasibility of learning from a mix of both fully-labeled supervised data and contextual bandit data.
Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
We study the sample complexity of model-based reinforcement learning (henceforth RL) in general contextual decision processes that require strategic exploration to find a near-optimal policy.
A Reductions Approach to Fair Classification
We present a systematic approach for achieving fairness in a binary classification setting.
Practical Contextual Bandits with Regression Oracles
A major challenge in contextual bandits is to design general-purpose algorithms that are both practically useful and theoretically well-founded.
On Oracle-Efficient PAC RL with Rich Observations
We study the computational tractability of PAC reinforcement learning with rich observations.
Hierarchical Imitation and Reinforcement Learning
We study how to effectively leverage expert feedback to learn sequential decision-making policies.
Learning Data-Driven Objectives to Optimize Interactive Systems
Effective optimization is essential for interactive systems to provide a satisfactory user experience.
A Contextual Bandit Bake-off
Contextual bandit algorithms are essential for solving many real-world interactive machine learning problems.
Efficient Contextual Bandits in Non-stationary Worlds
In this work, we develop several efficient contextual bandit algorithms for non-stationary environments by equipping existing methods for i. i. d.
Active Learning for Cost-Sensitive Classification
We design an active learning algorithm for cost-sensitive multiclass classification: problems where different errors have different costs.
Corralling a Band of Bandit Algorithms
We study the problem of combining multiple bandit algorithms (that is, online learning algorithms with partial feedback) with the goal of creating a master algorithm that performs almost as well as the best base algorithm if it were to be run on its own.
Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
We study the off-policy evaluation problem---estimating the value of a target policy using data collected by another policy---under the contextual bandit model.
Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
Our first contribution is a complexity measure, the Bellman rank, that we show enables tractable learning of near-optimal behavior in these processes and is naturally small for many well-studied reinforcement learning settings.
Making Contextual Decisions with Low Technical Debt
The Decision Service enables all aspects of contextual bandit learning using four system abstractions which connect together in a loop: explore (the decision space), log, learn, and deploy.
Off-policy evaluation for slate recommendation
This paper studies the evaluation of policies that recommend an ordered set of items (e. g., a ranking) based on some context---a common scenario in web search, ads, and recommendation.
Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains
We address both of these challenges with two complementary techniques: First, we develop a gradient-boosting style, non-parametric function approximator for learning on $Q$-function residuals.
PAC Reinforcement Learning with Rich Observations
We prove that the algorithm learns near optimal behavior after a number of episodes that is polynomial in all relevant parameters, logarithmic in the number of policies, and independent of the size of the observation space.
Efficient Second Order Online Learning by Sketching
We propose Sketched Online Newton (SON), an online second order learning algorithm that enjoys substantially improved regret guarantees for ill-conditioned data.
Fast Convergence of Regularized Learning in Games
We show that natural classes of regularized learning algorithms with a form of recency bias achieve faster convergence rates to approximate efficiency and to coarse correlated equilibria in multiplayer normal form games.
Efficient and Parsimonious Agnostic Active Learning
We develop a new active learning algorithm for the streaming setting satisfying three important properties: 1) It provably works for any classifier representation and classification problem including those with severe noise.
Contextual Semibandits via Supervised Learning Oracles
We study an online decision making problem where on each round a learner chooses a list of items based on some side information, receives a scalar feedback value for each individual item, and a reward that is linearly related to this feedback.
Learning to Search Better Than Your Teacher
Methods for learning to search for structured prediction typically imitate a reference policy, with existing theoretical guarantees demonstrating low regret compared to that reference.
Scalable Non-linear Learning with Adaptive Polynomial Expansions
Can we effectively learn a nonlinear representation in time comparable to linear learning?
A Lower Bound for the Optimization of Finite Sums
This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $\mu$-strongly convex.
Scalable Nonlinear Learning with Adaptive Polynomial Expansions
Can we effectively learn a nonlinear representation in time comparable to linear learning?
Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
We present a new algorithm for the contextual bandit learning problem, where the learner repeatedly takes one of $K$ actions in response to the observed context, and observes the reward only for that chosen action.
Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization
Alternating minimization is a popular heuristic for sparse coding, where the dictionary and the coefficients are estimated in alternate steps, keeping the other fixed.
Para-active learning
We leverage the same observation to build a generic strategy for parallelizing learning algorithms.
Least Squares Revisited: Scalable Approaches for Multi-class Prediction
This work provides simple algorithms for multi-class (and multi-label) prediction in settings where both the number of examples n and the data dimension d are relatively large.
A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries
We consider the problem of learning overcomplete dictionaries in the context of sparse coding, where each sample selects a sparse subset of dictionary elements.
Stochastic optimization and sparse statistical recovery: Optimal algorithms for high dimensions
We develop and analyze stochastic optimization algorithms for problems in which the expected loss is strongly convex, and the optimum is (approximately) sparse.
Distributed Delayed Stochastic Optimization
We analyze the convergence of gradient-based optimization algorithms whose updates depend on delayed stochastic gradient information.
A Reliable Effective Terascale Linear Learning System
We present a system and a set of techniques for learning linear predictors with convex losses on terascale datasets, with trillions of features, {The number of features here refers to the number of non-zero entries in the data matrix.}
Stochastic convex optimization with bandit feedback
This paper addresses the problem of minimizing a convex, Lipschitz function $f$ over a convex, compact set $\xset$ under a stochastic bandit feedback model.
Fast global convergence rates of gradient methods for high-dimensional statistical recovery
Many statistical $M$-estimators are based on convex optimization problems formed by the weighted sum of a loss function with a norm-based regularizer.
Distributed Dual Averaging In Networks
The goal of decentralized optimization over a network is to optimize a global objective formed by a sum of local (possibly nonsmooth) convex functions using only local computation and communication.
Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling
The goal of decentralized optimization over a network is to optimize a global objective formed by a sum of local (possibly nonsmooth) convex functions using only local computation and communication.
Information-theoretic lower bounds on the oracle complexity of convex optimization
The extensive use of convex optimization in machine learning and statistics makes such an understanding critical to understand fundamental computational limits of learning and estimation.
