Search Results for author:
Found 71 papers, 9 papers with code
Misspecified $Q$-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Error
In order to study whether the analog result is possible in the reinforcement learning setting, we consider the following problem: assuming the optimal $Q$-function is a $d$-dimensional linear function with sparsity $k$ and misspecification error $\epsilon$, whether we can obtain an $O\left(\epsilon\right)$-optimal policy using number of samples polynomially in the feature dimension $d$.
Learning for Bandits under Action Erasures
We consider a novel multi-arm bandit (MAB) setup, where a learner needs to communicate the actions to distributed agents over erasure channels, while the rewards for the actions are directly available to the learner through external sensors.
Confident Natural Policy Gradient for Local Planning in $q_π$-realizable Constrained MDPs
The constrained Markov decision process (CMDP) framework emerges as an important reinforcement learning approach for imposing safety or other critical objectives while maximizing cumulative reward.
Don't Forget to Connect! Improving RAG with Graph-based Reranking
Retrieval Augmented Generation (RAG) has greatly improved the performance of Large Language Model (LLM) responses by grounding generation with context from existing documents.
Multi-Agent Bandit Learning through Heterogeneous Action Erasure Channels
To our knowledge, these are the first algorithms capable of effectively learning through heterogeneous action erasure channels.
Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation
To tackle long planning horizon problems in reinforcement learning with general function approximation, we propose the first algorithm, termed as UCRL-WVTR, that achieves both \emph{horizon-free} and \emph{instance-dependent}, since it eliminates the polynomial dependency on the planning horizon.
Adaptive Liquidity Provision in Uniswap V3 with Deep Reinforcement Learning
The DRL policy aims to optimize trading fees earned by LPs against associated costs, such as gas fees and hedging expenses, which is referred to as loss-versus-rebalancing (LVR).
Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing
Our research demonstrates that to achieve $\epsilon$-optimal policies for all $M$ tasks, a single agent using DistMT-LSVI needs to run a total number of episodes that is at most $\tilde{\mathcal{O}}({d^3H^6(\epsilon^{-2}+c_{\rm sep}^{-2})}\cdot M/N)$, where $c_{\rm sep}>0$ is a constant representing task separability, $H$ is the horizon of each episode, and $d$ is the feature dimension of the dynamics and rewards.
Tackling Heavy-Tailed Rewards in Reinforcement Learning with Function Approximation: Minimax Optimal and Instance-Dependent Regret Bounds
Our algorithm, termed as \textsc{Heavy-LSVI-UCB}, achieves the \emph{first} computationally efficient \emph{instance-dependent} $K$-episode regret of $\tilde{O}(d \sqrt{H \mathcal{U}^*} K^\frac{1}{1+\epsilon} + d \sqrt{H \mathcal{V}^* K})$.
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models
In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to VL domain?
Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning
We provide an active-learning-based RL algorithm that first explores the environment without specifying a reward function and then asks a human teacher for only a few queries about the rewards of a task at some state-action pairs.
Does Sparsity Help in Learning Misspecified Linear Bandits?
In particular, Du et al. (2020) show that even if a learner is given linear features in $\mathbb{R}^d$ that approximate the rewards in a bandit or RL with a uniform error of $\varepsilon$, searching for an $O(\varepsilon)$-optimal action requires pulling at least $\Omega(\exp(d))$ queries.
Near Sample-Optimal Reduction-based Policy Learning for Average Reward MDP
This work considers the sample complexity of obtaining an $\varepsilon$-optimal policy in an average reward Markov Decision Process (AMDP), given access to a generative model (simulator).
Contexts can be Cheap: Solving Stochastic Contextual Bandits with Linear Bandit Algorithms
When the context distribution is unknown, we establish an algorithm that reduces the stochastic contextual instance to a sequence of linear bandit instances with small misspecifications and achieves nearly the same worst-case regret bound as the algorithm that solves the misspecified linear bandit instances.
Near-Optimal Sample Complexity Bounds for Constrained MDPs
In particular, we design a model-based algorithm that addresses two settings: (i) relaxed feasibility, where small constraint violations are allowed, and (ii) strict feasibility, where the output policy is required to satisfy the constraint.
Learning in Distributed Contextual Linear Bandits Without Sharing the Context
Contextual linear bandits is a rich and theoretically important model that has many practical applications.
Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximation
We study lifelong reinforcement learning (RL) in a regret minimization setting of linear contextual Markov decision process (MDP), where the agent needs to learn a multi-task policy while solving a streaming sequence of tasks.
Distributed Contextual Linear Bandits with Minimax Optimal Communication Cost
In particular, for scenarios with known context distribution, the communication cost of DisBE-LUCB is only $\tilde{\mathcal{O}}(dN)$ and its regret is ${\tilde{\mathcal{O}}}(\sqrt{dNT})$, which is of the same order as that incurred by an optimal single-agent algorithm for $NT$ rounds.
Solving Multi-Arm Bandit Using a Few Bits of Communication
Existing works usually fail to address this issue and can become infeasible in certain applications.
Settling the Horizon-Dependence of Sample Complexity in Reinforcement Learning
Notably, for an RL environment with horizon length $H$, previous work have shown that there is a probably approximately correct (PAC) algorithm that learns an $O(1)$-optimal policy using $\mathrm{polylog}(H)$ episodes of environment interactions when the number of states and actions is fixed.
Breaking the Moments Condition Barrier: No-Regret Algorithm for Bandits with Super Heavy-Tailed Payoffs
Despite a large amount of effort in dealing with heavy-tailed error in machine learning, little is known when moments of the error can become non-existential: the random noise $\eta$ satisfies Pr$\left[|\eta| > |y|\right] \le 1/|y|^{\alpha}$ for some $\alpha > 0$.
On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning
Multi-agent reinforcement learning (MARL) algorithms often suffer from an exponential sample complexity dependence on the number of agents, a phenomenon known as \emph{the curse of multiagents}.
Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation
To mitigate these issues, we propose a theoretically principled nearest neighbor (NN) function approximator that can improve the value networks in deep RL methods.
Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver
In particular, we take a plug-in solver approach, where we focus on learning a model in the exploration phase and demand that \emph{any planning algorithm} on the learned model can give a near-optimal policy.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Gap-Dependent Unsupervised Exploration for Reinforcement Learning
In particular, for an unknown finite-horizon Markov decision process, the algorithm takes only $\widetilde{\mathcal{O}} (1/\epsilon \cdot (H^3SA / \rho + H^4 S^2 A) )$ episodes of exploration, and is able to obtain an $\epsilon$-optimal policy for a post-revealed reward with sub-optimality gap at least $\rho$, where $S$ is the number of states, $A$ is the number of actions, and $H$ is the length of the horizon, obtaining a nearly \emph{quadratic saving} in terms of $\epsilon$.
Randomized Exploration for Reinforcement Learning with General Value Function Approximation
We propose a model-free reinforcement learning algorithm inspired by the popular randomized least squares value iteration (RLSVI) algorithm as well as the optimism principle.
Online Sub-Sampling for Reinforcement Learning with General Function Approximation
For a value-based method with complexity-bounded function class, we show that the policy only needs to be updated for $\propto\operatorname{poly}\log(K)$ times for running the RL algorithm for $K$ episodes while still achieving a small near-optimal regret bound.
Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphs
This global cache allows in-GPU importance sampling of mini-batches, which drastically reduces the number of nodes in a mini-batch, especially in the input layer, to reduce data copy between CPU and GPU and mini-batch computation without compromising the training convergence rate or model accuracy.
Safe Reinforcement Learning with Linear Function Approximation
Safety in reinforcement learning has become increasingly important in recent years.
Provably Correct Optimization and Exploration with Non-linear Policies
Policy optimization methods remain a powerful workhorse in empirical Reinforcement Learning (RL), with a focus on neural policies that can easily reason over complex and continuous state and/or action spaces.
Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
We establish an upper bound $O(|\mathcal{S}|H^{3/2}/N)$ for the suboptimality using the Mimic-MD algorithm in Rajaraman et al (2020) which we prove to be computationally efficient.
A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Cost
This paper focuses on the linear Markov Decision Process (MDP) recently studied in [Yang et al 2019, Jin et al 2020] where the linear function approximation is used for generalization on the large state space.
Minimax Sample Complexity for Turn-based Stochastic Game
The empirical success of Multi-agent reinforcement learning is encouraging, while few theoretical guarantees have been revealed.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning
We formalize this problem as an episodic learning problem on a Markov decision process, where transitions are unknown and a reward function is the inner product of a preference vector with pre-specified multi-objective reward functions.
Multi-Objective Reinforcement Learning
reinforcement-learning
Towards Fundamental Limits of Multi-armed Bandits with Random Walk Feedback
In this paper, we consider a new Multi-Armed Bandit (MAB) problem where arms are nodes in an unknown and possibly changing graph, and the agent (i) initiates random walks over the graph by pulling arms, (ii) observes the random walk trajectories, and (iii) receives rewards equal to the lengths of the walks.
Episodic Linear Quadratic Regulators with Low-rank Transitions
Consequently, the sample complexity of our algorithm only depends on the rank, $m$, rather than the ambient dimension, $d$, which can be orders-of-magnitude larger.
Is Plug-in Solver Sample-Efficient for Feature-based Reinforcement Learning?
However, the understanding of the sample optimality of model-based RL is still largely missing, even for the linear case.
Toward the Fundamental Limits of Imitation Learning
Here, we show that the policy which mimics the expert whenever possible is in expectation $\lesssim \frac{|\mathcal{S}| H^2 \log (N)}{N}$ suboptimal compared to the value of the expert, even when the expert follows an arbitrary stochastic policy.
Obtaining Adjustable Regularization for Free via Iterate Averaging
In sum, we obtain adjustable regularization for free for a large class of optimization problems and resolve an open question raised by Neu and Rosasco.
Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
This is in contrast to the usual reward-aware setting, with a $\tilde\Omega(|S|(|A|+|B|)(1-\gamma)^{-3}\epsilon^{-2})$ lower bound, where this model-based approach is near-optimal with only a gap on the $|A|,|B|$ dependence.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
On Reward-Free Reinforcement Learning with Linear Function Approximation
The sample complexity of our algorithm is polynomial in the feature dimension and the planning horizon, and is completely independent of the number of states and actions.
Preference-based Reinforcement Learning with Finite-Time Guarantees
If preferences are stochastic, and the preference probability relates to the hidden reward values, we present algorithms for PbRL, both with and without a simulator, that are able to identify the best policy up to accuracy $\varepsilon$ with high probability.
$Q$-learning with Logarithmic Regret
This paper presents the first non-asymptotic result showing that a model-free algorithm can achieve a logarithmic cumulative regret for episodic tabular reinforcement learning if there exists a strictly positive sub-optimality gap in the optimal $Q$-function.
Model-Based Reinforcement Learning with Value-Targeted Regression
We propose a model based RL algorithm that is based on optimism principle: In each episode, the set of models that are `consistent' with the data collected is constructed.
Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
Value function approximation has demonstrated phenomenal empirical success in reinforcement learning (RL).
Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?
Our analysis introduces two ideas: (i) the construction of an $\varepsilon$-net for optimal policies whose log-covering number scales only logarithmically with the planning horizon, and (ii) the Online Trajectory Synthesis algorithm, which adaptively evaluates all policies in a given policy class using sample complexity that scales with the log-covering number of the given policy class.
Provably Efficient Exploration for Reinforcement Learning Using Unsupervised Learning
Motivated by the prevailing paradigm of using unsupervised learning for efficient exploration in reinforcement learning (RL) problems [tang2017exploration, bellemare2016unifying], we investigate when this paradigm is provably efficient.
Sketching Transformed Matrices with Applications to Natural Language Processing
We show that our approach obtains small error and is efficient in both space and time.
Deep Reinforcement Learning with Linear Quadratic Regulator Regions
In this paper we propose a novel method that guarantees a stable region of attraction for the output of a policy trained in simulation, even for highly nonlinear systems.
How Does an Approximate Model Help in Reinforcement Learning?
In particular, we provide an algorithm that uses $\widetilde{O}(N/(1-\gamma)^3/\varepsilon^2)$ samples in a generative model to learn an $\varepsilon$-optimal policy, where $\gamma$ is the discount factor and $N$ is the number of near-optimal actions in the approximate model.
Continuous Control with Contexts, Provably
To our knowledge, this is first provably efficient algorithm to build a decoder in the continuous control setting.
Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
With regards to the statistical viewpoint, this question is largely unexplored, and the extant body of literature mainly focuses on conditions which permit sample efficient reinforcement learning with little understanding of what are necessary conditions for efficient reinforcement learning.
Efficient Symmetric Norm Regression via Linear Sketching
When the loss function is a general symmetric norm, our algorithm produces a $\sqrt{d} \cdot \mathrm{polylog} n \cdot \mathrm{mmc}(\ell)$-approximate solution in input-sparsity time, where $\mathrm{mmc}(\ell)$ is a quantity related to the symmetric norm under consideration.
PROVABLY BENEFITS OF DEEP HIERARCHICAL RL
Modern complex sequential decision-making problem often both low-level policy and high-level planning.
Solving Discounted Stochastic Two-Player Games with Near-Optimal Time and Sample Complexity
In this paper, we settle the sampling complexity of solving discounted two-player turn-based zero-sum stochastic games up to polylogarithmic factors.
Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal
In this work, we study the effectiveness of the most natural plug-in approach to model-based planning: we build the maximum likelihood estimate of the transition model in the MDP from observations and then find an optimal policy in this empirical MDP.
Model-based Reinforcement Learning
reinforcement-learning
+2
Feature-Based Q-Learning for Two-Player Stochastic Games
Consider a two-player zero-sum stochastic game where the transition function can be embedded in a given feature space.
Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
In this case, the kernelized MatrixRL satisfies a regret bound ${O}\big(H^2\widetilde{d}\log T\sqrt{T}\big)$, where $\widetilde{d}$ is the effective dimension of the kernel space.
Learning to Control in Metric Space with Optimal Regret
We study online reinforcement learning for finite-horizon deterministic control systems with {\it arbitrary} state and action spaces.
Sample-Optimal Parametric Q-Learning Using Linearly Additive Features
Consider a Markov decision process (MDP) that admits a set of state-action features, which can linearly express the process's probabilistic transition model.
Towards a Theoretical Understanding of Hashing-Based Neural Nets
In this paper, we provide provable guarantees on some hashing-based parameter reduction methods in neural nets.
On Landscape of Lagrangian Functions and Stochastic Search for Constrained Nonconvex Optimization
However, due to the lack of convexity, their landscape is not well understood and how to find the stable equilibria of the Lagrangian function is still unknown.
Near-Optimal Time and Sample Complexities for Solving Discounted Markov Decision Process with a Generative Model
In this paper we consider the problem of computing an $\epsilon$-optimal policy of a discounted Markov Decision Process (DMDP) provided we can only access its transition function through a generative sampling model that given any state-action pair samples from the transition function in $O(1)$ time.
Optimization and Control
Variance Reduction Methods for Sublinear Reinforcement Learning
There is a technical issue in the analysis that is not easily fixable.
Nearly Optimal Dynamic $k$-Means Clustering for High-Dimensional Data
We consider the $k$-means clustering problem in the dynamic streaming setting, where points from a discrete Euclidean space $\{1, 2, \ldots, \Delta\}^d$ can be dynamically inserted to or deleted from the dataset.
Misspecified Nonconvex Statistical Optimization for Phase Retrieval
Existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying "true" statistical models.
Online Partial Least Square Optimization: Dropping Convexity for Better Efficiency and Scalability
Multiview representation learning is popular for latent factor analysis.
On Quadratic Convergence of DC Proximal Newton Algorithm for Nonconvex Sparse Learning in High Dimensions
We propose a DC proximal Newton algorithm for solving nonconvex regularized sparse learning problems in high dimensions.
Online Factorization and Partition of Complex Networks From Random Walks
We formulate this into a nonconvex stochastic factorization problem and propose an efficient and scalable stochastic generalized Hebbian algorithm.
Dropping Convexity for More Efficient and Scalable Online Multiview Learning
Multiview representation learning is very popular for latent factor analysis.
The Physical Systems Behind Optimization Algorithms
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning.
