Search Results for author:
Found 66 papers, 9 papers with code
A Tutorial on Thompson Sampling
Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance.
Epistemic Neural Networks
We introduce the epinet: an architecture that can supplement any conventional neural network, including large pretrained models, and can be trained with modest incremental computation to estimate uncertainty.
The Neural Testbed: Evaluating Joint Predictions
Predictive distributions quantify uncertainties ignored by point estimates.
Evaluating High-Order Predictive Distributions in Deep Learning
Previous work has developed methods for assessing low-order predictive distributions with inputs sampled i. i. d.
Approximate Thompson Sampling via Epistemic Neural Networks
Further, we demonstrate that the \textit{epinet} -- a small additive network that estimates uncertainty -- matches the performance of large ensembles at orders of magnitude lower computational cost.
Online Influence Maximization under Independent Cascade Model with Semi-Bandit Feedback
Specifically, we aim to learn the set of "best influencers" in a social network online while repeatedly interacting with it.
Cascading Bandits for Large-Scale Recommendation Problems
In this work, we study cascading bandits, an online learning variant of the cascade model where the goal is to recommend $K$ most attractive items from a large set of $L$ candidate items.
Generalization and Exploration via Randomized Value Functions
We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions.
Offline Evaluation of Ranking Policies with Click Models
We analyze our estimators and prove that they are more efficient than the estimators that do not use the structure of the click model, under the assumption that the click model holds.
Deep Exploration via Randomized Value Functions
We study the use of randomized value functions to guide deep exploration in reinforcement learning.
Conservative Exploration using Interleaving
In many practical problems, a learning agent may want to learn the best action in hindsight without ever taking a bad action, which is significantly worse than the default production action.
Model-Independent Online Learning for Influence Maximization
We consider influence maximization (IM) in social networks, which is the problem of maximizing the number of users that become aware of a product by selecting a set of "seed" users to expose the product to.
New Insights into Bootstrapping for Bandits
We investigate the use of bootstrapping in the bandit setting.
SpectralLeader: Online Spectral Learning for Single Topic Models
We study the problem of learning a latent variable model from a stream of data.
Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Bandit
Multi-armed bandit (MAB) is a class of online learning problems where a learning agent aims to maximize its expected cumulative reward while repeatedly selecting to pull arms with unknown reward distributions.
Stochastic Low-Rank Bandits
Many problems in computer vision and recommender systems involve low-rank matrices.
Posterior Sampling for Large Scale Reinforcement Learning
Our algorithm termed deterministic schedule PSRL (DS-PSRL) is efficient in terms of time, sample, and space complexity.
Online Learning to Rank in Stochastic Click Models
In this work, we propose BatchRank, the first online learning to rank algorithm for a broad class of click models.
Does Weather Matter? Causal Analysis of TV Logs
To the best of our knowledge, this is the first large-scale causal study of the impact of weather on TV watching patterns.
Bernoulli Rank-$1$ Bandits for Click Feedback
The probability that a user will click a search result depends both on its relevance and its position on the results page.
Stochastic Rank-1 Bandits
The main challenge of the problem is that the individual values of the row and column are unobserved.
Efficient Learning in Large-Scale Combinatorial Semi-Bandits
A stochastic combinatorial semi-bandit is an online learning problem where at each step a learning agent chooses a subset of ground items subject to combinatorial constraints, and then observes stochastic weights of these items and receives their sum as a payoff.
Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization
We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization.
DCM Bandits: Learning to Rank with Multiple Clicks
This work presents the first practical and regret-optimal online algorithm for learning to rank with multiple clicks in a cascade-like click model.
Combinatorial Cascading Bandits
The agent observes the index of the first chosen item whose weight is zero.
Cascading Bandits: Learning to Rank in the Cascade Model
We also prove gap-dependent upper bounds on the regret of these algorithms and derive a lower bound on the regret in cascading bandits.
Tight Regret Bounds for Stochastic Combinatorial Semi-Bandits
A stochastic combinatorial semi-bandit is an online learning problem where at each step a learning agent chooses a subset of ground items subject to constraints, and then observes stochastic weights of these items and receives their sum as a payoff.
Learning to Act Greedily: Polymatroid Semi-Bandits
Many important optimization problems, such as the minimum spanning tree and minimum-cost flow, can be solved optimally by a greedy method.
DUM: Diversity-Weighted Utility Maximization for Recommendations
The need for diversification of recommendation lists manifests in a number of recommender systems use cases.
Optimal Demand Response Using Device Based Reinforcement Learning
Demand response (DR) for residential and small commercial buildings is estimated to account for as much as 65% of the total energy savings potential of DR, and previous work shows that a fully automated Energy Management System (EMS) is a necessary prerequisite to DR in these areas.
Matroid Bandits: Fast Combinatorial Optimization with Learning
The objective in these problems is to learn how to maximize a modular function on a matroid.
Profit Maximization for Online Advertising Demand-Side Platforms
We develop an optimization model and corresponding algorithm for the management of a demand-side platform (DSP), whereby the DSP aims to maximize its own profit while acquiring valuable impressions for its advertiser clients.
Optimization and Control Computer Science and Game Theory
Online Diverse Learning to Rank from Partial-Click Feedback
We assume that the user examines the list of recommended items until the user is attracted by an item, which is clicked, and does not examine the rest of the items.
Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed Bandits
Specifically, it pulls the arm with the highest mean reward in a non-parametric bootstrap sample of its history with pseudo rewards.
Scalar Posterior Sampling with Applications
Our algorithm termed deterministic schedule PSRL (DS-PSRL) is efficient in terms of time, sample, and space complexity.
Adaptive Submodular Maximization in Bandit Setting
Maximization of submodular functions has wide applications in machine learning and artificial intelligence.
Efficient Exploration and Value Function Generalization in Deterministic Systems
We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization.
Scalable Thompson Sampling via Optimal Transport
Thompson sampling (TS) is a class of algorithms for sequential decision-making, which requires maintaining a posterior distribution over a model.
Stochastic Online Learning with Probabilistic Graph Feedback
We consider a problem of stochastic online learning with general probabilistic graph feedback, where each directed edge in the feedback graph has probability $p_{ij}$.
Waterfall Bandits: Learning to Sell Ads Online
We design an online learning algorithm for solving this problem, which interleaves learning and optimization, and prove that this algorithm has sublinear regret.
Bootstrapping Upper Confidence Bound
Upper Confidence Bound (UCB) method is arguably the most celebrated one used in online decision making with partial information feedback.
Nested-Wasserstein Self-Imitation Learning for Sequence Generation
Reinforcement learning (RL) has been widely studied for improving sequence-generation models.
Improving Adversarial Text Generation by Modeling the Distant Future
Auto-regressive text generation models usually focus on local fluency, and may cause inconsistent semantic meaning in long text generation.
Hypermodels for Exploration
This generalizes and extends the use of ensembles to approximate Thompson sampling.
Influence Diagram Bandits: Variational Thompson Sampling for Structured Bandit Problems
We propose a novel framework for structured bandits, which we call an influence diagram bandit.
Structured Policy Iteration for Linear Quadratic Regulator
In this paper, we introduce the \textit{Structured Policy Iteration} (S-PI) for LQR, a method capable of deriving a structured linear policy.
Stochastic Low-rank Tensor Bandits for Multi-dimensional Online Decision Making
We consider two settings, tensor bandits without context and tensor bandits with context.
On the Sample Complexity of Reinforcement Learning with Policy Space Generalization
To avoid such undesirable dependence on the state and action space sizes, this paper proposes a new notion of eluder dimension for the policy space, which characterizes the intrinsic complexity of policy learning in an arbitrary Markov Decision Process (MDP).
Influence Diagram Bandits
We experiment with three structured bandit problems: cascading bandits, online learning to rank in the position-based model, and rank-1 bandits.
Budgeted Online Influence Maximization
We introduce a new budgeted framework for online influence maximization, considering the total cost of an advertising campaign instead of the common cardinality constraint on a chosen influencer set.
A Benchmark and Baseline for Language-Driven Image Editing
To solve this new task, we first present a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations.
On Efficiency in Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) approaches promise to provide more efficient solutions to sequential decision making problems, both in terms of statistical as well as computational efficiency.
Neural Contextual Bandits with Deep Representation and Shallow Exploration
We study a general class of contextual bandits, where each context-action pair is associated with a raw feature vector, but the reward generating function is unknown.
On the Approximation Relationship between Optimizing Ratio of Submodular (RS) and Difference of Submodular (DS) Functions
We demonstrate that from an algorithm guaranteeing an approximation factor for the ratio of submodular (RS) optimization problem, we can build another algorithm having a different kind of approximation guarantee -- weaker than the classical one -- for the difference of submodular (DS) optimization problem, and vice versa.
Data Structures and Algorithms
Reinforcement Learning, Bit by Bit
To illustrate concepts, we design simple agents that build on them and present computational results that highlight data efficiency.
Joint Online Learning and Decision-making via Dual Mirror Descent
We consider an online revenue maximization problem over a finite time horizon subject to lower and upper bounds on cost.
From Predictions to Decisions: The Importance of Joint Predictive Distributions
A fundamental challenge for any intelligent system is prediction: given some inputs, can you predict corresponding outcomes?
Evaluating Predictive Distributions: Does Bayesian Deep Learning Work?
This paper introduces \textit{The Neural Testbed}, which provides tools for the systematic evaluation of agents that generate such predictions.
An Analysis of Ensemble Sampling
Ensemble sampling serves as a practical approximation to Thompson sampling when maintaining an exact posterior distribution over model parameters is computationally intractable.
Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping
In machine learning, an agent needs to estimate uncertainty to efficiently explore and adapt and to make effective decisions.
Robustness of Epinets against Distributional Shifts
However, these improvements are relatively small compared to the outstanding issues in distributionally-robust deep learning.
Leveraging Demonstrations to Improve Online Learning: Quality Matters
This offers insight into how pretraining can greatly improve online performance and how the degree of improvement increases with the expert's competence level.
Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale
We first propose an Informed Posterior Sampling-based RL (iPSRL) algorithm that uses the offline dataset, and information about the expert's behavioral policy used to generate the offline dataset.
Active learning for effective Hamiltonian of super-large-scale atomic structures
However, the parametrization method of the effective Hamiltonian is complicated and hardly can resolve the systems with complex interactions and/or complex components.
Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach
In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with.
RLHF and IIA: Perverse Incentives
Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA).
