Search Results for author:
Found 85 papers, 16 papers with code
Adaptive Sampling for Estimating Probability Distributions
We consider the problem of allocating a fixed budget of samples to a finite set of discrete distributions to learn them uniformly well (minimizing the maximum error) in terms of four common distance measures: $\ell_2^2$, $\ell_1$, $f$-divergence, and separation distance.
Aligning Text-to-Image Models using Human Feedback
Our results demonstrate the potential for learning from human feedback to significantly improve text-to-image models.
Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management
By exploiting MoE-LM structure, our methods significantly reduce the size of the action space and improve the efficacy of RL-based DM.
Multi-Task Off-Policy Learning from Bandit Feedback
We prove per-task bounds on the suboptimality of the learned policies, which show a clear improvement over not using the hierarchical model.
Operator Splitting Value Iteration
We introduce new planning and reinforcement learning algorithms for discounted MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function.
RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk
We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and discounted infinite-horizon MDPs.
Robust Reinforcement Learning using Offline Data
The goal of robust reinforcement learning (RL) is to learn a policy that is robust against the uncertainty in model parameters.
Reinforcement Learning of Multi-Domain Dialog Policies Via Action Embeddings
Learning task-oriented dialog policies via reinforcement learning typically requires large amounts of interaction with users, which in practice renders such methods unusable for real-world applications.
A Mixture-of-Expert Approach to RL-based Dialogue Management
Despite recent advancements in language models (LMs), their application to dialogue management (DM) problems and ability to carry on rich conversations remain a challenge.
Multi-Environment Meta-Learning in Stochastic Linear Bandits
In this work we investigate meta-learning (or learning-to-learn) approaches in multi-task linear stochastic bandit problems that can originate from multiple environments.
Collaborative Multi-agent Stochastic Linear Bandits
We propose a distributed upper confidence bound (UCB) algorithm and prove a high probability bound on its $T$-round regret in which we include a linear growth of regret associated with each communication round.
Efficient Risk-Averse Reinforcement Learning
In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns.
Non-stationary Bandits and Meta-Learning with a Small Set of Optimal Arms
We study a sequential decision problem where the learner faces a sequence of $K$-armed bandit tasks.
Meta-Learning for Simple Regret Minimization
This is while we show that the meta simple regret of the frequentist algorithm is $\tilde{O}(\sqrt{m} n + m/ \sqrt{n})$, and thus, worse.
Deep Hierarchy in Bandits
We use this exact posterior to analyze the Bayes regret of HierTS in Gaussian bandits.
Hierarchical Bayesian Bandits
We provide a unified view of all these problems, as learning to act in a hierarchical Bayesian bandit.
Thompson Sampling with a Mixture Prior
We study Thompson sampling (TS) in online decision making, where the uncertain environment is sampled from a mixture distribution.
Fixed-Budget Best-Arm Identification in Structured Bandits
We analyze our algorithm in linear and generalized linear models (GLMs) and propose a practical implementation based on a G-optimal design.
Feature and Parameter Selection in Stochastic Linear Bandits
In the second setting, the reward parameter of the LB problem is arbitrarily selected from $M$ models represented as (possibly) overlapping balls in $\mathbb R^d$.
Adaptive Sampling for Minimax Fair Classification
Machine learning models trained on uncurated datasets can often end up adversely affecting inputs belonging to underrepresented groups.
Non-Stationary Latent Bandits
The key idea is to frame this problem as a latent bandit, where the prototypical models of user behavior are learned offline and the latent state of the user is inferred online from its interactions with the models.
Soft-Robust Algorithms for Batch Reinforcement Learning
In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the percentile criterion, which minimizes the probability of a catastrophic failure.
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
Uncertainty quantification (UQ) plays a pivotal role in reduction of uncertainties during both optimization and decision making processes.
Variance-Reduced Off-Policy Memory-Efficient Policy Search
To achieve variance-reduced off-policy-stable policy optimization, we propose an algorithm family that is memory-efficient, stochastically variance-reduced, and capable of learning from off-policy samples.
Deep Bayesian Quadrature Policy Optimization
On the other hand, more sample efficient alternatives like Bayesian quadrature methods have received little attention due to their high computational complexity.
Control-Aware Representations for Model-based Reinforcement Learning
We first formulate a LCE model to learn representations that are suitable to be used by a policy iteration style algorithm in the latent space.
Model-based Reinforcement Learning
reinforcement-learning
+2
Stochastic Bandits with Linear Constraints
We propose an upper-confidence bound algorithm for this problem, called optimistic pessimistic linear bandit (OPLB), and prove an $\widetilde{\mathcal{O}}(\frac{d\sqrt{T}}{\tau-c_0})$ bound on its $T$-round regret, where the denominator is the difference between the constraint threshold and the cost of a known feasible action.
Variational Model-based Policy Optimization
Model-based reinforcement learning (RL) algorithms allow us to combine model-generated data with those collected from interaction with the real system in order to alleviate the data efficiency problem in RL.
Neural Lyapunov Redesign
We provide theoretical results on the class of systems that can be treated with the proposed algorithm and empirically evaluate the effectiveness of our method using an exemplary dynamical system.
Finite-Sample Analysis of Proximal Gradient TD Algorithms
In this paper, we analyze the convergence rate of the gradient temporal difference learning (GTD) family of algorithms.
Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning with Polynomial Sample Complexity
In this paper, we introduce proximal gradient temporal difference learning, which provides a principled way of designing and analyzing true stochastic gradient temporal difference learning algorithms.
Mirror Descent Policy Optimization
Overall, MDPO is derived from the MD principles, offers a unified approach to viewing a number of popular RL algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a number of continuous control tasks.
Active Model Estimation in Markov Decision Processes
Using a number of simple domains with heterogeneous noise in their transitions, we show that our heuristic-based algorithm outperforms both our original algorithm and the maximum entropy algorithm in the small sample regime, while achieving similar asymptotic performance as that of the original algorithm.
Predictive Coding for Locally-Linear Control
High-dimensional observations and unknown dynamics are major challenges when applying optimal control to many real-world decision making tasks.
Policy-Aware Model Learning for Policy Gradient Methods
This is in contrast to conventional model learning approaches, such as those based on maximum likelihood estimate, that learn a predictive model of the environment without explicitly considering the interaction of the model and the planner.
Conservative Exploration in Reinforcement Learning
While learning in an unknown Markov Decision Process (MDP), an agent should trade off exploration to discover new information about the MDP, and exploitation of the current knowledge to maximize the reward.
Improved Algorithms for Conservative Exploration in Bandits
In this case, it is desirable to deploy online learning algorithms (e. g., a multi-armed bandit algorithm) that interact with the system to learn a better/optimal policy under the constraint that during the learning process the performance is almost never worse than the performance of the baseline itself.
Adaptive Sampling for Estimating Multiple Probability Distributions
We consider the problem of allocating samples to a finite set of discrete distributions in order to learn them uniformly well in terms of four common distance measures: $\ell_2^2$, $\ell_1$, $f$-divergence, and separation distance.
Multi-step Greedy Reinforcement Learning Algorithms
We derive model-free RL algorithms based on $\kappa$-PI and $\kappa$-VI in which the surrogate problem can be solved by any discrete or continuous action RL method, such as DQN and TRPO.
Benchmarking Batch Deep Reinforcement Learning Algorithms
Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment.
Safe Policy Learning for Continuous Control
We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i. e.,~policies that keep the agent in desirable situations, both during training and at convergence.
Multi-step Greedy Policies in Model-Free Deep Reinforcement Learning
In this work, we explore the benefits of multi-step greedy policies in model-free RL when employed in the framework of multi-step Dynamic Programming (DP): multi-step Policy and Value Iteration.
Online Planning with Lookahead Policies
This is the first work that proves improved sample complexity as a result of {\em increasing} the lookahead horizon in online planning.
Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
A promising approach is to embed the high-dimensional observations into a lower-dimensional latent representation space, estimate the latent dynamics model, then utilize this model for control in the latent space.
Randomized Exploration in Generalized Linear Bandits
GLM-TSL samples a generalized linear model (GLM) from the Laplace approximation to the posterior distribution.
Active Learning for Binary Classification with Abstention
We construct and analyze active learning algorithms for the problem of binary classification with abstention.
Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policies
In this paper, we focus on model-based RL in the finite-state finite-horizon MDP setting and establish that exploring with \emph{greedy policies} -- act by \emph{1-step planning} -- can achieve tight minimax performance in terms of regret, $\tilde{\mathcal{O}}(\sqrt{HSAT})$.
Model-based Reinforcement Learning
reinforcement-learning
+1
Binary Classification with Bounded Abstention Rate
We then propose a plug-in classifier that employs unlabeled samples to decide the region of abstention and derive an upper-bound on the excess risk of our classifier under standard \emph{H\"older smoothness} and \emph{margin} assumptions.
Perturbed-History Exploration in Stochastic Linear Bandits
Our algorithm, perturbed-history exploration in a linear bandit (LinPHE), estimates a linear model from its perturbed history and pulls the arm with the highest value under that model.
Perturbed-History Exploration in Stochastic Multi-Armed Bandits
Finally, we empirically evaluate PHE and show that it is competitive with state-of-the-art baselines.
Lyapunov-based Safe Policy Optimization for Continuous Control
We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them.
Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed Bandits
Specifically, it pulls the arm with the highest mean reward in a non-parametric bootstrap sample of its history with pseudo rewards.
Lyapunov-based Safe Policy Optimization
In many reinforcement learning applications, it is crucial that the agent interacts with the environment only through safe policies, i. e.,~policies that do not take the agent to certain undesirable situations.
A Block Coordinate Ascent Algorithm for Mean-Variance Optimization
Risk management in dynamic decision problems is a primary concern in many fields, including financial investment, autonomous driving, and healthcare.
Risk-Sensitive Generative Adversarial Imitation Learning
We then derive two different versions of our RS-GAIL optimization problem that aim at matching the risk profiles of the agent and the expert w. r. t.
A Lyapunov-based Approach to Safe Reinforcement Learning
In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints.
Optimizing over a Restricted Policy Class in Markov Decision Processes
However, under a condition that is akin to the occupancy measures of the base policies having large overlap, we show that there exists an efficient algorithm that finds a policy that is almost as good as the best convex combination of the base policies.
More Robust Doubly Robust Off-policy Evaluation
In particular, we focus on the doubly robust (DR) estimators that consist of an importance sampling (IS) component and a performance model, and utilize the low (or zero) bias of IS and low variance of the model at the same time.
Path Consistency Learning in Tsallis Entropy Regularized MDPs
In this paper, we follow the work of Nachum et al. (2017) in the soft ERL setting, and propose a class of novel path consistency learning (PCL) algorithms, called {\em sparse PCL}, for the sparse ERL problem that can work with both on-policy and off-policy data.
Disentangling Dynamics and Content for Control and Planning
In this paper, We study the problem of learning a controllable representation for high-dimensional observations of dynamical systems.
Robust Locally-Linear Controllable Embedding
We also propose a principled variational approximation of the embedding posterior that takes the future observation into account, and thus, makes the variational approximation more robust against the noise.
Online Learning to Rank in Stochastic Click Models
In this work, we propose BatchRank, the first online learning to rank algorithm for a broad class of click models.
Active Learning for Accurate Estimation of Linear Models
We explore the sequential decision making problem where the goal is to estimate uniformly well a number of linear models, given a shared budget of random contexts independently sampled from a known distribution.
Model-Independent Online Learning for Influence Maximization
We consider influence maximization (IM) in social networks, which is the problem of maximizing the number of users that become aware of a product by selecting a set of "seed" users to expose the product to.
Bottleneck Conditional Density Estimation
We introduce a new framework for training deep generative models for high-dimensional conditional density estimation.
Conservative Contextual Linear Bandits
We prove an upper-bound on the regret of CLUCB and show that it can be decomposed into two terms: 1) an upper-bound for the regret of the standard linear UCB algorithm that grows with the time horizon and 2) a constant (does not grow with the time horizon) term that accounts for the loss of being conservative in order to satisfy the safety constraint.
Bayesian Reinforcement Learning: A Survey
The objective of the paper is to provide a comprehensive survey on Bayesian RL algorithms and their theoretical and empirical properties.
Safe Policy Improvement by Minimizing Robust Baseline Regret
We show that our formulation is NP-hard and propose an approximate algorithm.
Personalized Advertisement Recommendation: A Ranking Approach to Address the Ubiquitous Click Sparsity Problem
We study the problem of personalized advertisement recommendation (PAR), which consist of a user visiting a system (website) and the system displaying one of $K$ ads to the user.
Graphical Model Sketch
Graphical models are a popular approach to modeling structured data but they are unsuitable for high-cardinality variables.
Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i. e., increased awareness of events of small probability and high consequences.
Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits
If the variance of the distributions were known, one could design an optimal sampling strategy by collecting a number of independent samples per distribution that is proportional to their variance.
Policy Gradient for Coherent Risk Measures
For static risk measures, our approach is in the spirit of policy gradient algorithms and combines a standard sampling approach with convex programming.
Classification-based Approximate Policy Iteration: Experiments and Extended Discussions
We introduce a general classification-based approximate policy iteration (CAPI) framework, which encompasses a large class of algorithms that can exploit regularities of both the value function and the policy space, depending on what is advantageous.
Algorithms for CVaR Optimization in MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion.
Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPs
For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize.
Approximate Dynamic Programming Finally Performs Well in the Game of Tetris
A close look at the literature of this game shows that while ADP algorithms, that have been (almost) entirely based on approximating the value function (value function based), have performed poorly in Tetris, the methods that search directly in the space of policies by learning the policy parameters using an optimization black box, such as the cross entropy (CE) method, have achieved the best reported results.
Actor-Critic Algorithms for Risk-Sensitive MDPs
For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize.
Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence
We study the problem of identifying the best arm(s) in the stochastic multi-armed bandit setting.
Approximate Modified Policy Iteration
Modified policy iteration (MPI) is a dynamic programming (DP) algorithm that contains the two celebrated policy and value iteration methods.
A Generalized Kernel Approach to Structured Output Learning
Finally, we evaluate the performance of our KDE approach using both covariance and conditional covariance kernels on two structured output problems, and compare it to the state-of-the-art kernel-based structured output regression methods.
Multi-Bandit Best Arm Identification
We first propose an algorithm called Gap-based Exploration (GapE) that focuses on the arms whose mean is close to the mean of the best arm in the same bandit (i. e., small gap).
Speedy Q-Learning
We introduce a new convergent variant of Q-learning, called speedy Q-learning, to address the problem of slow convergence in the standard form of the Q-learning algorithm.
LSTD with Random Projections
We provide a thorough theoretical analysis of the LSTD with random projections and derive performance bounds for the resulting algorithm.
Regularized Policy Iteration
In this paper we consider approximate policy-iteration-based reinforcement learning algorithms.
