Search Results for author:
Found 63 papers, 8 papers with code
Provable Interactive Learning with Hindsight Instruction Feedback
We study interactive learning in a setting where the agent has to generate a response (e. g., an action or trajectory) given a context and an instruction.
Provably Sample Efficient RLHF via Active Preference Optimization
Experimental evaluations on a human preference dataset validate \texttt{APO}'s efficacy as a sample-efficient and practical solution to data collection for RLHF, facilitating alignment of LLMs with human preferences in a cost-effective and scalable manner.
A Framework for Partially Observed Reward-States in RLHF
We show reductions from the the two dominant forms of human feedback in RLHF - cardinal and dueling feedback to PORRL.
Contextual Bandits with Stage-wise Constraints
In the setting that the constraint is in expectation, we further specialize our results to multi-armed bandits and propose a computationally efficient algorithm for this setting with regret analysis.
Experiment Planning with Function Approximation
We study the problem of experiment planning with function approximation in contextual bandit problems.
Unbiased Decisions Reduce Regret: Adversarial Domain Adaptation for the Bank Loan Problem
We introduce adversarial optimism (AdOpt) to directly address bias in the training set using adversarial domain adaptation.
Anytime Model Selection in Linear Bandits
This allows us to develop ALEXP, which has an exponentially improved ($\log M$) dependence on $M$ for its regret.
Data-Driven Online Model Selection With Regret Guarantees
We consider model selection for sequential decision making in stochastic environments with bandit feedback, where a meta-learner has at its disposal a pool of base learners, and decides on the fly which action to take based on the policies recommended by each base learner.
Improving Offline RL by Blending Heuristics
We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping.
Estimating Optimal Policy Value in General Linear Contextual Bandits
Whether this is possible for more realistic context distributions has remained an open and important question for tasks such as model selection.
Transfer RL via the Undo Maps Formalism
Transferring knowledge across domains is one of the most fundamental problems in machine learning, but doing so effectively in the context of reinforcement learning remains largely an open problem.
Leveraging Offline Data in Online Reinforcement Learning
Practical scenarios often motivate an intermediate setting: if we have some set of offline data and, in addition, may also interact with the environment, how can we best use the offline data to minimize the number of online interactions necessary to learn an $\epsilon$-optimal policy?
Learning General World Models in a Handful of Reward-Free Deployments
We then present CASCADE, a novel approach for self-supervised exploration in this new setting.
Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity
Reinforcement learning provides an automated framework for learning behaviors from high-level reward specifications, but in practice the choice of reward function can be crucial for good results -- while in principle the reward only needs to specify what the task is, in reality practitioners often need to design more detailed rewards that provide the agent with some hints about how the task should be completed.
Neural Design for Genetic Perturbation Experiments
The problem of how to genetically modify cells in order to maximize a certain cellular phenotype has taken center stage in drug development over the last few years (with, for example, genetically edited CAR-T, CAR-NK, and CAR-NKT cells entering cancer clinical trials).
Best of Both Worlds Model Selection
We study the problem of model selection in bandit scenarios in the presence of nested policy classes, with the goal of obtaining simultaneous adversarial and stochastic ("best of both worlds") high-probability regret guarantees.
Joint Representation Training in Sequential Tasks with Shared Structure
In contrast with previous work that have studied multi task RL in other function approximation models, we show that in the presence of bilinear optimization oracle and finite state action spaces there exists a computationally efficient algorithm for multitask MatrixRL via a reduction to quadratic programming.
Online Nonsubmodular Minimization with Delayed Costs: From Full Information to Bandit Feedback
Motivated by applications to online learning in sparse estimation and Bayesian optimization, we consider the problem of online unconstrained nonsubmodular minimization with delayed costs in both full information and bandit feedback settings.
Meta Learning MDPs with Linear Transition Models
We study meta-learning in Markov Decision Processes (MDP) with linear transition models in the undiscounted episodic setting.
Neural Pseudo-Label Optimism for the Bank Loan Problem
The lender only observes whether a customer will repay a loan if the loan is issued to begin with, and thus modeled decisions affect what data is available to the lender for future decisions.
Dueling RL: Reinforcement Learning with Trajectory Preferences
We consider the problem of preference based reinforcement learning (PbRL), where, unlike traditional reinforcement learning, an agent receives feedback only in terms of a 1 bit (0/1) preference over a trajectory pair instead of absolute rewards for them.
An Instance-Dependent Analysis for the Cooperative Multi-Player Multi-Armed Bandit
We study the problem of information sharing and cooperation in Multi-Player Multi-Armed bandits.
Towards an Understanding of Default Policies in Multitask Policy Optimization
Much of the recent success of deep reinforcement learning has been driven by regularized policy optimization (RPO) algorithms with strong performance across multiple domains.
Reinforcement Learning in Linear MDPs: Constant Regret and Representation Selection
We study the role of the representation of state-action value functions in regret minimization in finite-horizon Markov Decision Processes (MDPs) with linear structure.
Sample Efficient Reinforcement Learning In Continuous State Spaces: A Perspective Beyond Linearity
Reinforcement learning (RL) is empirically successful in complex nonlinear Markov decision processes (MDPs) with continuous state spaces.
On the Theory of Reinforcement Learning with Once-per-Episode Feedback
We study a theory of reinforcement learning (RL) in which the learner receives binary feedback only once at the end of an episode.
Parallelizing Contextual Bandits
Standard approaches to decision-making under uncertainty focus on sequential exploration of the space of decisions.
Near Optimal Policy Optimization via REPS
Since its introduction a decade ago, \emph{relative entropy policy search} (REPS) has demonstrated successful policy learning on a number of simulated and real-world robotic domains, not to mention providing algorithmic components used by many recently proposed reinforcement learning (RL) algorithms.
Unlocking Pixels for Reinforcement Learning via Implicit Attention
There has recently been significant interest in training reinforcement learning (RL) agents in vision-based environments.
Tactical Optimism and Pessimism for Deep Reinforcement Learning
In recent years, deep off-policy actor-critic algorithms have become a dominant approach to reinforcement learning for continuous control.
ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
In this paper, we approach the problem of optimizing blackbox functions over large hybrid search spaces consisting of both combinatorial and continuous parameters.
Fairness with Continuous Optimal Transport
Whilst optimal transport (OT) is increasingly being recognized as a powerful and flexible approach for dealing with fairness issues, current OT fairness methods are confined to the use of discrete OT.
Regret Bound Balancing and Elimination for Model Selection in Bandits and RL
Finally, unlike recent efforts in model selection for linear stochastic bandits, our approach is versatile enough to also cover cases where the context information is generated by an adversarial environment, rather than a stochastic one.
Online Model Selection for Reinforcement Learning with Function Approximation
Towards this end, we consider the problem of model selection in RL with function approximation, given a set of candidate RL algorithms with known regret guarantees.
Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
In the era of ever decreasing loss functions, SGD and its various offspring have become the go-to optimization tool in machine learning and are a key component of the success of deep neural networks (DNNs).
Accelerated Message Passing for Entropy-Regularized MAP Inference
Maximum a posteriori (MAP) inference in discrete-valued Markov random fields is a fundamental problem in machine learning that involves identifying the most likely configuration of random variables given a distribution.
Towards Tractable Optimism in Model-Based Reinforcement Learning
The principle of optimism in the face of uncertainty is prevalent throughout sequential decision making problems such as multi-armed bandits and reinforcement learning (RL).
Stochastic Bandits with Linear Constraints
We propose an upper-confidence bound algorithm for this problem, called optimistic pessimistic linear bandit (OPLB), and prove an $\widetilde{\mathcal{O}}(\frac{d\sqrt{T}}{\tau-c_0})$ bound on its $T$-round regret, where the denominator is the difference between the constraint threshold and the cost of a known feasible action.
Taming the Herd: Multi-Modal Meta-Learning with a Population of Agents
Meta-learning is a paradigm whereby an agent is trained with the specific goal of fast adaptation.
Regret Balancing for Bandit and RL Model Selection
Given a set of base learning algorithms, an effective model selection strategy adapts to the best learning algorithm in an online fashion.
Learning the Truth From Only One Side of the Story
Learning under one-sided feedback (i. e., where we only observe the labels for examples we predicted positively on) is a fundamental problem in machine learning -- applications include lending and recommendation systems.
Stochastic Flows and Geometric Optimization on the Orthogonal Group
We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group $O(d)$ and naturally reductive homogeneous manifolds obtained from the action of the rotation group $SO(d)$.
Robustness Guarantees for Mode Estimation with an Application to Bandits
Mode estimation is a classical problem in statistics with a wide range of applications in machine learning.
Model Selection in Contextual Stochastic Bandit Problems
Our methods rely on a novel and generic smoothing transformation for bandit algorithms that permits us to obtain optimal $O(\sqrt{T})$ model selection guarantees for stochastic contextual bandit problems as long as the optimal base algorithm satisfies a high probability regret guarantee.
On Thompson Sampling with Langevin Algorithms
The resulting approximate Thompson sampling algorithm has logarithmic regret and its computational complexity does not scale with the time horizon of the algorithm.
Ready Policy One: World Building Through Active Learning
Model-Based Reinforcement Learning (MBRL) offers a promising direction for sample efficient learning, often achieving state of the art results for continuous control tasks.
Effective Diversity in Population Based Reinforcement Learning
Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment.
Reinforcement Learning with Chromatic Networks
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
ES-MAML: Simple Hessian-Free Meta Learning
We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES).
Behavior-Guided Reinforcement Learning
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
Wasserstein Fair Classification
We propose an approach to fair classification that enforces independence between the classifier outputs and sensitive information by minimizing Wasserstein-1 distances.
Reinforcement Learning with Chromatic Networks for Compact Architecture Search
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
Convergence Rates of Smooth Message Passing with Rounding in Entropy-Regularized MAP Inference
Maximum a posteriori (MAP) inference is a fundamental computational paradigm for statistical inference.
Learning to Score Behaviors for Guided Policy Optimization
We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space.
Structured Monte Carlo Sampling for Nonisotropic Distributions via Determinantal Point Processes
We propose a new class of structured methods for Monte Carlo (MC) sampling, called DPPMC, designed for high-dimensional nonisotropic distributions where samples are correlated to reduce the variance of the estimator via determinantal point processes.
Provably Robust Blackbox Optimization for Reinforcement Learning
Interest in derivative-free optimization (DFO) and "evolutionary strategies" (ES) has recently surged in the Reinforcement Learning (RL) community, with growing evidence that they can match state of the art methods for policy optimization problems in Robotics.
From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
ASEBO adapts to the geometry of the function and learns optimal sets of sensing directions, which are used to probe it, on-the-fly.
Geometrically Coupled Monte Carlo Sampling
Monte Carlo sampling in high-dimensional, low-sample settings is important in many machine learning tasks.
Gen-Oja: Simple & Efficient Algorithm for Streaming Generalized Eigenvector Computation
In this paper, we study the problems of principle Generalized Eigenvector computation and Canonical Correlation Analysis in the stochastic setting.
Gen-Oja: A Two-time-scale approach for Streaming CCA
In this paper, we study the problems of principal Generalized Eigenvector computation and Canonical Correlation Analysis in the stochastic setting.
Online learning with kernel losses
We also study the full information setting when the underlying losses are kernel functions and present an adapted exponential weights algorithm and a conditional gradient descent algorithm.
Reinforcement Learning with Wasserstein Distance Regularisation, with Applications to Multipolicy Learning
We describe an application of Wasserstein distance to Reinforcement Learning.
Real time clustering of time series using triangular potentials
Motivated by the problem of computing investment portfolio weightings we investigate various methods of clustering as alternatives to traditional mean-variance approaches.
