Search Results for author:
Found 75 papers, 18 papers with code
Explicit Recall for Efficient Exploration
In this paper, we advocate the use of explicit memory for efficient exploration in reinforcement learning.
MESOB: Balancing Equilibria & Social Optimality
Motivated by bid recommendation in online ad auctions, this paper considers a general class of multi-level and multi-agent games, with two major characteristics: one is a large number of anonymous agents, and the other is the intricate interplay between competition and cooperation.
Offline Policy Optimization in RL with Variance Regularizaton
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications.
A Reinforcement Learning Approach to Estimating Long-term Treatment Effects
Randomized experiments (a. k. a.
Understanding Domain Randomization for Sim-to-real Transfer
Reinforcement learning encounters many challenges when applied directly in the real world.
A Map of Bandits for E-commerce
The rich body of Bandit literature not only offers a diverse toolbox of algorithms, but also makes it hard for a practitioner to find the right solution to solve the problem at hand.
On the Optimality of Batch Policy Optimization Algorithms
First, we introduce a class of confidence-adjusted index algorithms that unifies optimistic and pessimistic principles in a common framework, which enables a general analysis.
Near-optimal Representation Learning for Linear Bandits and Linear RL
This paper studies representation learning for multi-task linear bandits and multi-task episodic RL with linear value function approximation.
Offline Policy Optimization with Variance Regularization
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications.
Escaping the Gravitational Pull of Softmax
Both findings are based on an analysis of convergence rates using the Non-uniform \L{}ojasiewicz (N\L{}) inequalities.
CoinDICE: Off-Policy Confidence Interval Estimation
We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies.
Neural Thompson Sampling
Thompson Sampling (TS) is one of the most effective algorithms for solving contextual multi-armed bandit problems.
Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RL
The regret of FMDP-BF is shown to be exponentially smaller than that of optimal algorithms designed for non-factored MDPs, and improves on the best previous result for FMDPs~\citep{osband2014near} by a factored of $\sqrt{H|\mathcal{S}_i|}$, where $|\mathcal{S}_i|$ is the cardinality of the factored state subspace and $H$ is the planning horizon.
Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders.
Off-Policy Evaluation via the Regularized Lagrangian
The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data.
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible.
Batch Stationary Distribution Estimation
We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions.
GenDICE: Generalized Offline Estimation of Stationary Values
An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain.
Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing
Deep Reinforcement Learning (RL) is proven powerful for decision making in simulated environments.
AlgaeDICE: Policy Gradient from Arbitrary Experience
In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility.
Neural Contextual Bandits with UCB-based Exploration
To the best of our knowledge, it is the first neural network-based contextual bandit algorithm with a near-optimal regret guarantee.
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
Our method is doubly robust in that the bias vanishes when either the density ratio or the value function estimation is perfect.
NeuralUCB: Contextual Bandits with Neural Network-Based Exploration
To the best of our knowledge, our algorithm is the first neural network-based contextual bandit algorithm with near-optimal regret guarantee.
Randomized Exploration in Generalized Linear Bandits
The first, GLM-TSL, samples a generalized linear model (GLM) from the Laplace approximation to the posterior distribution.
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
In contrast to previous approaches, our algorithm is agnostic to knowledge of the behavior policy (or policies) used to generate the dataset.
A Kernel Loss for Solving the Bellman Equation
Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms.
Neural Logic Machines
We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning.
Policy Certificates: Towards Accountable Reinforcement Learning
The performance of a reinforcement learning algorithm can vary drastically during learning because of exploration.
Adversarial Attacks on Stochastic Bandits
We study adversarial attacks that manipulate the reward signals to control the actions chosen by a stochastic multi-armed bandit algorithm.
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy.
Neural Approaches to Conversational AI
The present paper surveys neural approaches to conversational AI that have been developed in the last few years.
Data Poisoning Attacks in Contextual Bandits
We provide a general attack framework based on convex optimization and show that by slightly manipulating rewards in the data, an attacker can force the bandit algorithm to pull a target arm for a target contextual vector.
Scalable Bilinear Pi Learning Using State and Action Features
In this work, we study a primal-dual formulation of the ALP, and develop a scalable, model-free algorithm called bilinear $\pi$ learning for reinforcement learning when a sampling oracle is provided.
Scalable Bilinear $π$ Learning Using State and Action Features
In this work, we study a primal-dual formulation of the ALP, and develop a scalable, model-free algorithm called bilinear $\pi$ learning for reinforcement learning when a sampling oracle is provided.
Subgoal Discovery for Hierarchical Dialogue Policy Learning
Experiments with simulated and real users show that our approach performs competitively against a state-of-the-art method that requires human-defined subgoals.
Avoiding Catastrophic States with Intrinsic Fear
Many practical reinforcement learning problems contain catastrophic states that the optimal policy visits infrequently or never.
Now I Remember! Episodic Memory For Reinforcement Learning
Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car.
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
When function approximation is used, solving the Bellman optimality equation with stability guarantees has remained a major open problem in reinforcement learning for decades.
Boosting the Actor with Dual Critic
This paper proposes a new actor-critic-style algorithm called Dual Actor-Critic or Dual-AC.
Q-LDA: Uncovering Latent Patterns in Text-based Sequential Decision Processes
In sequential decision making, it is often important and useful for end users to understand the underlying patterns or causes that lead to the corresponding decisions.
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems.
Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks.
Investigation of Language Understanding Impact for Reinforcement Learning Based Dialogue Systems
Language understanding is a key component in a spoken dialogue system.
End-to-End Task-Completion Neural Dialogue Systems
One of the major drawbacks of modularized task-completion dialogue systems is that each module is trained individually, which presents several challenges.
Scaffolding Networks: Incremental Learning and Teaching Through Questioning
We introduce a new paradigm of learning for reasoning, understanding, and prediction, as well as the scaffolding network to implement this paradigm.
Provably Optimal Algorithms for Generalized Linear Contextual Bandits
Contextual bandits are widely used in Internet services from news recommendation to advertising, and to Web search.
Stochastic Variance Reduction Methods for Policy Evaluation
Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy.
A User Simulator for Task-Completion Dialogues
Then, one can train reinforcement learning agents in an online fashion as they interact with the simulator.
Active Learning with Oracle Epiphany
We present a theoretical analysis of active learning with more realistic interactions with human oracles.
Neuro-Symbolic Program Synthesis
While achieving impressive results, these approaches have a number of important limitations: (a) they are computationally expensive and hard to train, (b) a model has to be trained for each task (program) separately, and (c) it is hard to interpret or verify the correctness of the learnt mapping (as it is defined by a neural network).
Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes.
Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access
In this paper, we address this limitation by replacing symbolic queries with an induced "soft" posterior distribution over the KB that indicates which entities the user is interested in.
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems.
Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads
We introduce an online popularity prediction and tracking task as a benchmark task for reinforcement learning with a combinatorial, natural language action space.
Deep Reinforcement Learning with a Natural Language Action Space
This paper introduces a novel architecture for reinforcement learning with deep neural networks designed to handle state and action spaces characterized by natural language, as found in text-based games.
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy.
Recurrent Reinforcement Learning: A Hybrid Approach
Successful applications of reinforcement learning in real-world problems often require dealing with partially observable states.
The Online Coupon-Collector Problem and Its Application to Lifelong Reinforcement Learning
Transferring knowledge across a sequence of related tasks is an important challenge in reinforcement learning (RL).
An efficient algorithm for contextual bandits with knapsacks, and an extension to concave objectives
This problem was introduced by Badanidiyuru et al. (2014), who gave a computationally inefficient algorithm with near-optimal regret bounds for it.
On the Prior Sensitivity of Thompson Sampling
The empirically successful Thompson Sampling algorithm for stochastic bandits has drawn much interest in understanding its theoretical properties.
Evaluation of Explore-Exploit Policies in Multi-result Ranking Systems
An offline framework is thus necessary to let us decide what policy and how we should apply in a production environment to ensure positive outcome.
Doubly Robust Policy Evaluation and Optimization
As such, we expect the doubly robust approach to become common practice in policy evaluation and optimization.
On Minimax Optimal Offline Policy Evaluation
This paper studies the off-policy evaluation problem, where one aims to estimate the value of a target policy based on a sample of observations collected by another policy.
Counterfactual Estimation and Optimization of Click Metrics for Search Engines
Optimizing an interactive system against a predefined online metric is particularly challenging, when the metric is computed from user feedback such as clicks and payments.
Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
We present a new algorithm for the contextual bandit learning problem, where the learner repeatedly takes one of $K$ actions in response to the observed context, and observes the reward only for that chosen action.
Efficient Online Bootstrapping for Large Scale Learning
Bootstrapping is a useful technique for estimating the uncertainty of a predictor, for example, confidence intervals for prediction.
Generalized Thompson Sampling for Contextual Bandits
Similar to most expert-learning algorithms, Generalized Thompson Sampling uses a loss function to adjust the experts' weights.
Sample Complexity of Multi-task Reinforcement Learning
Transferring knowledge across a sequence of reinforcement-learning tasks is challenging, and has a number of important applications.
An Empirical Evaluation of Thompson Sampling
Thompson sampling is one of oldest heuristic to address the exploration / exploitation trade-off, but it is surprisingly not very popular in the literature.
Doubly Robust Policy Evaluation and Learning
The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy.
Parallelized Stochastic Gradient Descent
With the increase in available data parallel machine learning has become an increasingly pressing problem.
Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
\emph{Offline} evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "partial-label" nature.
A Contextual-Bandit Approach to Personalized News Article Recommendation
In this work, we model personalized recommendation of news articles as a contextual bandit problem, a principled approach in which a learning algorithm sequentially selects articles to serve users based on contextual information about the users and articles, while simultaneously adapting its article-selection strategy based on user-click feedback to maximize total user clicks.
Learning from Logged Implicit Exploration Data
We provide a sound and consistent foundation for the use of \emph{nonrandom} exploration data in "contextual bandit" or "partially labeled" settings where only the value of a chosen action is learned.
Sparse Online Learning via Truncated Gradient
We propose a general method called truncated gradient to induce sparsity in the weights of online-learning algorithms with convex loss.
