Search Results for author:
Found 79 papers, 7 papers with code
Regret Minimization via Saddle Point Optimization
A long line of works characterizes the sample complexity of regret minimization in sequential decision-making by min-max programs.
Switching the Loss Reduces the Cost in Batch Reinforcement Learning
We propose training fitted Q-iteration with log-loss (FQI-LOG) for batch reinforcement learning (RL).
Ensemble sampling for linear bandits: small ensembles suffice
We provide the first useful and rigorous analysis of ensemble sampling for the stochastic linear bandit setting.
Exploration via linearly perturbed loss minimisation
We show that, for the case of generalised linear bandits, EVILL reduces to perturbed history exploration (PHE), a method where exploration is done by training on randomly perturbed rewards.
Stochastic Gradient Descent for Gaussian Processes Done Right
We study the optimisation problem associated with Gaussian process regression using squared loss.
Online RL in Linearly $q^π$-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignore
We consider online reinforcement learning (RL) in episodic Markov decision processes (MDPs) under the linear $q^\pi$-realizability assumption, where it is assumed that the action-values of all policies can be expressed as linear functions of state-action features.
The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation
Theoretical guarantees in reinforcement learning (RL) are known to suffer multiplicative blow-up factors with respect to the misspecification error of function approximation.
Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice
Mirror descent value iteration (MDVI), an abstraction of Kullback-Leibler (KL) and entropy-regularized reinforcement learning (RL), has served as the basis for recent high-performing practical RL algorithms.
Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL
While policy optimization algorithms have played an important role in recent empirical success of Reinforcement Learning (RL), the existing theoretical understanding of policy optimization remains rather limited -- they are either restricted to tabular MDPs or suffer from highly suboptimal sample complexity, especial in online RL where exploration is necessary.
Exponential Hardness of Reinforcement Learning with Linear Function Approximation
The rewards in this game are chosen such that if the learner achieves large reward, then the learner's actions can be used to simulate solving a variant of 3-SAT, where (a) each variable shows up in a bounded number of clauses (b) if an instance has no solutions then it also has no solutions that satisfy more than (1-$\epsilon$)-fraction of clauses.
Efficient Planning in Combinatorial Action Spaces with Applications to Cooperative Multi-Agent Reinforcement Learning
A practical challenge in reinforcement learning are combinatorial action spaces that make planning computationally demanding.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networks
We explore the ability of overparameterized shallow ReLU neural networks to learn Lipschitz, nondifferentiable, bounded functions with additive noise when trained by Gradient Descent (GD).
Revisiting Simple Regret: Fast Rates for Returning a Good Arm
Simple regret is a natural and parameter-free performance criterion for pure exploration in multi-armed bandits yet is less popular than the probability of missing the best arm or an $\epsilon$-good arm, perhaps due to lack of easy ways to characterize it.
Confident Approximate Policy Iteration for Efficient Local Planning in $q^π$-realizable MDPs
Our first contribution is a new variant of Approximate Policy Iteration (API), called Confident Approximate Policy Iteration (CAPI), which computes a deterministic stationary policy with an optimal error bound scaling linearly with the product of the effective horizon $H$ and the worst-case approximation error $\epsilon$ of the action-value functions of stationary policies.
Optimistic MLE -- A Generic Model-based Algorithm for Partially Observable Sequential Decision Making
We prove that OMLE learns the near-optimal policies of an enormously rich class of sequential decision making problems in a polynomial number of samples.
Near-Optimal Sample Complexity Bounds for Constrained MDPs
In particular, we design a model-based algorithm that addresses two settings: (i) relaxed feasibility, where small constraint violations are allowed, and (ii) strict feasibility, where the output policy is required to satisfy the constraint.
Bandit Theory and Thompson Sampling-Guided Directed Evolution for Sequence Optimization
We propose a Thompson Sampling-guided Directed Evolution (TS-DE) framework for sequence optimization, where the sequence-to-function mapping is unknown and querying a single value is subject to costly and noisy measurements.
Sample-Efficient Reinforcement Learning of Partially Observable Markov Games
This paper considers the challenging tasks of Multi-Agent Reinforcement Learning (MARL) under partial observability, where each agent only sees her own individual observations and actions that reveal incomplete information about the underlying state of system.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal
In this work, we consider and analyze the sample complexity of model-free reinforcement learning with a generative model.
When Is Partially Observable Reinforcement Learning Not Scary?
Applications of Reinforcement Learning (RL), in which agents learn to make a sequence of decisions despite lacking complete information about the latent states of the controlled system, that is, they act under partial observability of the states, are ubiquitous.
Partially Observable Reinforcement Learning
reinforcement-learning
+1
A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
Representation learning lies at the heart of the empirical success of deep learning for dealing with the curse of dimensionality.
Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs
We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs).
TensorPlan and the Few Actions Lower Bound for Planning in MDPs under Linear Realizability of Optimal Value Functions
Furthermore, we show that the upper bound of TensorPlan can be extended to hold under (iii) and, for MDPs with deterministic transitions and stochastic rewards, also under (ii).
Efficient Local Planning with Linear Function Approximation
Under the assumption that the Q-functions of all policies are linear in known features of the state-action pairs, we show that our algorithms have polynomial query and computational costs in the dimension of the features, the effective planning horizon, and the targeted sub-optimality, while these costs are independent of the size of the state space.
On the Role of Optimization in Double Descent: A Least Squares Study
Empirically it has been observed that the performance of deep neural networks steadily improves as we increase model size, contradicting the classical view on overfitting and generalization.
No Regrets for Learning the Prior in Bandits
We propose ${\tt AdaTS}$, a Thompson sampling algorithm that adapts sequentially to bandit tasks that it interacts with.
Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping
We explore the ability of overparameterized shallow neural networks to learn Lipschitz regression functions with and without label noise when trained by Gradient Descent (GD).
Optimization Issues in KL-Constrained Approximate Policy Iteration
We compare the use of KL divergence as a constraint vs. as a regularizer, and point out several optimization issues with the widely-used constrained approach.
Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood.
On Query-efficient Planning in MDPs under Linear Realizability of the Optimal State-value Function
We consider local planning in fixed-horizon MDPs with a generative model under the assumption that the optimal value function lies close to the span of a feature map.
Asymptotically Optimal Information-Directed Sampling
We introduce a simple and efficient algorithm for stochastic linear bandits with finitely many actions that is asymptotically optimal and (nearly) worst-case optimal in finite time.
Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
To evaluate a new target policy, we analyze a Lasso fitted Q-evaluation method and establish a finite-sample error bound that has no polynomial dependence on the ambient dimension.
Online Sparse Reinforcement Learning
First, we provide a lower bound showing that linear regret is generally unavoidable in this case, even if there exists a policy that collects well-conditioned data.
A Distribution-Dependent Analysis of Meta-Learning
A key problem in the theory of meta-learning is to understand how the task distributions influence transfer risk, the expected error of a meta-learner on a new task drawn from the unknown task distribution.
Online Algorithm for Unsupervised Sequential Selection with Contextual Information
In this paper, we study Contextual Unsupervised Sequential Selection (USS), a new variant of the stochastic contextual bandits problem where the loss of an arm cannot be inferred from the observed feedback.
CoinDICE: Off-Policy Confidence Interval Estimation
We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies.
Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions
We consider the problem of local planning in fixed-horizon and discounted Markov Decision Processes (MDPs) with linear function approximation and a generative model under the assumption that the optimal action-value function lies in the span of a feature map that is available to the planner.
Tighter risk certificates for neural networks
In the context of probabilistic neural networks, the output of training is a probability distribution over network weights.
Efficient Planning in Large MDPs with Weak Linear Function Approximation
Large-scale Markov decision processes (MDPs) require planning algorithms with runtime independent of the number of states of the MDP.
Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting
We consider off-policy evaluation in the contextual bandit setting for the purpose of obtaining a robust off-policy selection strategy, where the selection strategy is evaluated based on the value of the chosen policy in a set of proposal (target) policies.
Efron-Stein PAC-Bayesian Inequalities
We prove semi-empirical concentration inequalities for random variables which are given as possibly nonlinear functions of independent random variables.
Detecting Overfitting via Adversarial Examples
It utilizes a new unbiased error estimate that is based on adversarial examples generated from the test data and importance weighting.
Distribution-Dependent Analysis of Gibbs-ERM Principle
This is a well-established notion of effective dimension appearing in several previous works, including the analyses of SGD and ridge regression, but ours is the first work that brings this dimension to the analysis of learning using Gibbs densities.
Online Algorithm for Unsupervised Sensor Selection
We set up the USS problem as a stochastic partial monitoring problem and develop an algorithm with sub-linear regret under the WD property.
Online Learning to Rank with Features
We introduce a new model for online ranking in which the click probability factors into an examination and attractiveness function and the attractiveness function is a linear function of a feature vector and an unknown parameter.
LeapsAndBounds: A Method for Approximately Optimal Algorithm Configuration
We consider the problem of configuring general-purpose solvers to run efficiently on problem instances drawn from an unknown distribution.
Linear Stochastic Approximation: Constant Step-Size and Iterate Averaging
For a given LSA with PR averaging, and data distribution $P$ satisfying the said assumptions, we show that there exists a range of constant step-sizes such that its MSE decays as $O(\frac{1}{t})$.
A Modular Analysis of Adaptive (Non-)Convex Optimization: Optimism, Composite Objectives, and Variational Bounds
Recently, much work has been done on extending the scope of online learning and incremental stochastic optimization algorithms.
Mixing time estimation in reversible Markov chains from a single sample path
The interval is constructed around the relaxation time $t_{\text{relax}} = 1/\gamma$, which is strongly related to the mixing time, and the width of the interval converges to zero roughly at a $1/\sqrt{n}$ rate, where $n$ is the length of the sample path.
Structured Best Arm Identification with Fixed Confidence
We study the problem of identifying the best action among a set of possible options when the value of each action is given by a mapping from a number of noisy micro-observables in the so-called fixed confidence setting.
Bernoulli Rank-$1$ Bandits for Click Feedback
The probability that a user will click a search result depends both on its relevance and its position on the results page.
Following the Leader and Fast Rates in Linear Prediction: Curved Constraint Sets and Other Regularities
The follow the leader (FTL) algorithm, perhaps the simplest of all online learning algorithms, is known to perform well when the loss functions it is used on are convex and positively curved.
SDP Relaxation with Randomized Rounding for Energy Disaggregation
We develop a scalable, computationally efficient method for the task of energy disaggregation for home appliance monitoring.
(Bandit) Convex Optimization with Biased Noisy Gradient Oracles
Algorithms for bandit convex optimization and online learning often rely on constructing noisy gradient estimates, which are then used in appropriately adjusted first-order algorithms, replacing actual gradients.
Multiclass Classification Calibration Functions
We devise a streamlined analysis that simplifies the process of deriving calibration functions for a large number of surrogate losses that have been proposed in the literature.
Chaining Bounds for Empirical Risk Minimization
This paper extends the standard chaining technique to prove excess risk upper bounds for empirical risk minimization with random design settings even if the magnitude of the noise and the estimates is unbounded.
Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models
In this paper we study a model-based approach to calculating approximately optimal policies in Markovian Decision Processes.
Model-based Reinforcement Learning
reinforcement-learning
+1
Conservative Bandits
We consider both the stochastic and the adversarial settings, where we propose, natural, yet novel strategies and analyze the price for maintaining the constraints.
DCM Bandits: Learning to Rank with Multiple Clicks
This work presents the first practical and regret-optimal online algorithm for learning to rank with multiple clicks in a cascade-like click model.
Online Learning with Gaussian Payoffs and Side Observations
For the first time in the literature, we provide non-asymptotic problem-dependent lower bounds on the regret of any algorithm, which recover existing asymptotic problem-dependent lower bounds and finite-time minimax lower bounds available in the literature.
Fast Cross-Validation for Incremental Learning
Cross-validation (CV) is one of the main tools for performance estimation and parameter tuning in machine learning.
Mixing Time Estimation in Reversible Markov Chains from a Single Sample Path
The interval is constructed around the relaxation time $t_{\text{relax}}$, which is strongly related to the mixing time, and the width of the interval converges to zero roughly at a $\sqrt{n}$ rate, where $n$ is the length of the sample path.
Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control
Cumulative prospect theory (CPT) is known to model human decisions well, with substantial empirical evidence supporting this claim.
Optimal Resource Allocation with Semi-Bandit Feedback
We study a sequential resource allocation problem involving a fixed number of recurring jobs.
Adaptive Monte Carlo via Bandit Allocation
We consider the problem of sequentially choosing between a set of unbiased Monte Carlo estimators to minimize the mean-squared-error (MSE) of a final combined estimate.
Online Learning under Delayed Feedback
Online learning with delayed feedback has received increasing attention recently due to its several applications in distributed, web-based learning problems.
Deep Representations and Codes for Image Auto-Annotation
The task of assigning a set of relevant tags to an image is challenging due to the size and variability of tag vocabularies.
Improved Algorithms for Linear Stochastic Bandits
We improve the theoretical analysis and empirical performance of algorithms for the stochastic multi-armed bandit problem and the linear stochastic multi-armed bandit problem.
Parametric Bandits: The Generalized Linear Case
We consider structured multi-armed bandit tasks in which the agent is guided by prior structural knowledge that can be exploited to efficiently select the optimal arm(s) in situations where the number of arms is large, or even infinite.
Online Markov Decision Processes under Bandit Feedback
We consider online learning in finite stochastic Markovian environments where in each time step a new reward function is chosen by an oblivious adversary.
Error Propagation for Approximate Policy and Value Iteration
We address the question of how the approximation error/Bellman residual at each iteration of the Approximate Policy/Value Iteration algorithms influences the quality of the resulted policy.
Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
We present simple and computationally efficient nonparametric estimators of R\'enyi entropy and mutual information based on an i. i. d.
A General Projection Property for Distribution Families
We prove that linear projections between distribution families with fixed first and second moments are surjective, regardless of dimension.
Multi-Step Dyna Planning for Policy Evaluation and Control
We extend Dyna planning architecture for policy evaluation and control in two significant aspects.
Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation
We introduce the first temporal-difference learning algorithms that converge with smooth value function approximators, such as neural networks.
A Convergent O(n) Temporal-difference Algorithm for Off-policy Learning with Linear Function Approximation
We introduce the first temporal-difference learning algorithm that is stable with linear function approximation and off-policy training, for any finite Markov decision process, target policy, and exciting behavior policy, and whose complexity scales linearly in the number of parameters.
Regularized Policy Iteration
In this paper we consider approximate policy-iteration-based reinforcement learning algorithms.
Online Optimization in X-Armed Bandits
We consider a generalization of stochastic bandit problems where the set of arms, X, is allowed to be a generic topological space.
Fitted Q-iteration in continuous action-space MDPs
We consider continuous state, continuous action batch reinforcement learning where the goal is to learn a good policy from a sufficiently rich trajectory generated by another policy.
