Search Results for author:
Found 177 papers, 26 papers with code
FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning
However, establishing high-quality market environments and benchmarks for financial reinforcement learning is challenging due to three major factors, namely, low signal-to-noise ratio of financial data, survivorship bias of historical data, and model overfitting in the backtesting stage.
Dynamic Datasets and Market Environments for Financial Reinforcement Learning
The financial market is a particularly challenging playground for deep reinforcement learning due to its unique feature of dynamic datasets.
ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning
In this paper, we present a scalable and elastic library ElegantRL-podracer for cloud-native deep reinforcement learning, which efficiently supports millions of GPU cores to carry out massively parallel training at multiple levels.
FinRL-Meta: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance
In this paper, we present a FinRL-Meta framework that builds a universe of market environments for data-driven financial reinforcement learning.
Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
This paper develops a Pontryagin Differentiable Programming (PDP) methodology, which establishes a unified framework to solve a broad class of learning and control tasks.
Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency
Specifically, we design a prompt template for reasoning that learns from the memory buffer and plans a future trajectory over a long horizon ("reason for future").
Provably Efficient Reinforcement Learning with Linear Function Approximation
Modern Reinforcement Learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy.
Randomized Exploration for Reinforcement Learning with General Value Function Approximation
We propose a model-free reinforcement learning algorithm inspired by the popular randomized least squares value iteration (RLSVI) algorithm as well as the optimism principle.
Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
This gives a deeper understanding of why the in-sample learning paradigm works, i. e., it applies implicit value regularization to the policy.
Sample Elicitation
We also show a connection between this sample elicitation problem and $f$-GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples.
Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
We show that such OOD sampling and pessimistic bootstrapping yields provable uncertainty quantifier in linear MDPs, thus providing the theoretical underpinning for PBRL.
RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks.
False Correlation Reduction for Offline Reinforcement Learning
Offline reinforcement learning (RL) harnesses the power of massive datasets for resolving sequential decision problems.
Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning
Moreover, under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration
To achieve this, existing sample-efficient online RL algorithms typically consist of three components: estimation, planning, and exploration.
Convergent Policy Optimization for Safe Reinforcement Learning
We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonconvex functions.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Principled Exploration via Optimistic Bootstrapping and Backward Induction
In this paper, we propose a principled exploration method for DRL through Optimistic Bootstrapping and Backward Induction (OB2I).
Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning
Motivated by the observation, we present a multi-agent PPO algorithm in which the local policy of each agent is updated similarly to vanilla PPO.
Dynamic Bottleneck for Robust Self-Supervised Exploration
Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards.
Differentiable Bilevel Programming for Stackelberg Congestion Games
A Stackelberg congestion game (SCG) is a bilevel program in which a leader aims to maximize their own gain by anticipating and manipulating the equilibrium state at which followers settle by playing a congestion game.
Pessimism Meets Invariance: Provably Efficient Offline Mean-Field Multi-Agent RL
Theoretically, under a weak coverage assumption that the experience dataset contains enough information about the optimal policy, we prove that for an episodic mean-field MDP with a horizon $H$ and $N$ training trajectories, SAFARI attains a sub-optimality gap of $\mathcal{O}(H^2d_{\rm eff} /\sqrt{N})$, where $d_{\rm eff}$ is the effective dimension of the function class for parameterizing the value function, but independent on the number of agents.
High-dimensional Varying Index Coefficient Models via Stein's Identity
We study the parameter estimation problem for a varying index coefficient model in high dimensions.
Exponential Family Model-Based Reinforcement Learning via Score Matching
We propose an optimistic model-based algorithm, dubbed SMRL, for finite-horizon episodic reinforcement learning (RL) when the transition model is specified by exponential family distributions with $d$ parameters and the reward is bounded and known.
A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy.
Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning.
On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond
We establish a margin based data dependent generalization error bound for a general family of deep neural networks in terms of the depth and width, as well as the Jacobian of the networks.
Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
Despite the success of single-agent reinforcement learning, multi-agent reinforcement learning (MARL) remains challenging due to complex interactions between agents.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Detecting Nonlinear Causality in Multivariate Time Series with Sparse Additive Models
We propose a nonparametric method for detecting nonlinear causal relationship within a set of multidimensional discrete time series, by using sparse additive models (SpAMs).
Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization
We propose a general theory for studying the \xl{landscape} of nonconvex \xl{optimization} with underlying symmetric structures \tz{for a class of machine learning problems (e. g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}.
Misspecified Nonconvex Statistical Optimization for Phase Retrieval
Existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying "true" statistical models.
NESTT: A Nonconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization
We study a stochastic and distributed algorithm for nonconvex problems whose objective consists of a sum of $N$ nonconvex $L_i/N$-smooth functions, plus a nonsmooth regularizer.
Tensor Graphical Model: Non-convex Optimization and Statistical Inference
We consider the estimation and inference of graphical models that characterize the dependency structure of high-dimensional tensor-valued data.
Sparse Generalized Eigenvalue Problem: Optimal Statistical Rates via Truncated Rayleigh Flow
Sparse generalized eigenvalue problem (GEP) plays a pivotal role in a large family of high-dimensional statistical models, including sparse Fisher's discriminant analysis, canonical correlation analysis, and sufficient dimension reduction.
Sharp Computational-Statistical Phase Transitions via Oracle Computational Model
Based upon an oracle model of computation, which captures the interactions between algorithms and data, we establish a general lower bound that explicitly connects the minimum testing risk under computational budget constraints with the intrinsic probabilistic and combinatorial structures of statistical problems.
Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference
To recover $\beta^*$, we propose an $\ell_1$-regularized least-squares estimator.
Optimal linear estimation under unknown nonlinear transform
This model is known as the single-index model in statistics, and, among other things, it represents a significant generalization of one-bit compressed sensing.
Statistical Limits of Convex Relaxations
Many high dimensional sparse learning problems are formulated as nonconvex optimization.
High Dimensional Expectation-Maximization Algorithm: Statistical Optimization and Asymptotic Normality
We provide a general theory of the expectation-maximization (EM) algorithm for inferring high dimensional latent variable models.
Optimal computational and statistical rates of convergence for sparse nonconvex learning problems
In particular, our analysis improves upon existing results by providing a more refined sample complexity bound as well as an exact support recovery result for the final estimator.
Nonconvex Statistical Optimization: Minimax-Optimal Sparse PCA in Polynomial Time
To optimally estimate sparse principal subspaces, we propose a two-stage computational framework named "tighten after relax": Within the 'relax' stage, we approximately solve a convex relaxation of sparse PCA with early stopping to obtain a desired initial estimator; For the 'tighten' stage, we propose a novel algorithm called sparse orthogonal iteration pursuit (SOAP), which iteratively refines the initial estimator by directly solving the underlying nonconvex problem.
Sparse Principal Component Analysis for High Dimensional Vector Autoregressive Models
We study sparse principal component analysis for high dimensional vector autoregressive time series under a doubly asymptotic framework, which allows the dimension $d$ to scale with the series length $T$.
Off-Policy Evaluation and Learning from Logged Bandit Feedback: Error Reduction via Surrogate Policy
Such an error reduction phenomenon is somewhat surprising as the estimated surrogate policy is less accurate than the given historical policy.
Curse of Heterogeneity: Computational Barriers in Sparse Mixture Models and Phase Retrieval
We study the fundamental tradeoffs between statistical accuracy and computational tractability in the analysis of high dimensional heterogeneous data.
Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processes
Despite the empirical success of nonconvex statistical optimization methods, their global dynamics, especially convergence to the desirable local minima, remain less well understood in theory.
A convex formulation for high-dimensional sparse sliced inverse regression
Sliced inverse regression is a popular tool for sufficient dimension reduction, which replaces covariates with a minimal set of their linear combinations without loss of information on the conditional distribution of the response given the covariates.
Provable Gaussian Embedding with One Observation
In this paper, we study the Gaussian embedding model and develop the first theoretical results for exponential family embedding models.
Blind Attacks on Machine Learners
We study the potential of a “blind attacker” to provably limit a learner’s performance by data injection attack without observing the learner’s training set or any parameter of the distribution from which it is drawn.
Agnostic Estimation for Misspecified Phase Retrieval Models
The goal of noisy high-dimensional phase retrieval is to estimate an $s$-sparse parameter $\boldsymbol{\beta}^*\in \mathbb{R}^d$ from $n$ realizations of the model $Y = (\boldsymbol{X}^{\top} \boldsymbol{\beta}^*)^2 + \varepsilon$.
A Nonconvex Optimization Framework for Low Rank Matrix Estimation
We study the estimation of low rank matrices via nonconvex optimization.
Non-convex Statistical Optimization for Sparse Tensor Graphical Model
We consider the estimation of sparse graphical models that characterize the dependency structure of high-dimensional tensor-valued data.
High Dimensional EM Algorithm: Statistical Optimization and Asymptotic Normality
We provide a general theory of the expectation-maximization (EM) algorithm for inferring high dimensional latent variable models.
Tighten after Relax: Minimax-Optimal Sparse PCA in Polynomial Time
In this paper, we propose a two-stage sparse PCA procedure that attains the optimal principal subspace estimator in polynomial time.
The Edge Density Barrier: Computational-Statistical Tradeoffs in Combinatorial Inference
We study the hypothesis testing problem of inferring the existence of combinatorial structures in undirected graphical models.
On Tighter Generalization Bounds for Deep Neural Networks: CNNs, ResNets, and Beyond
We propose a generalization error bound for a general family of deep neural networks based on the depth and width of the networks, as well as the spectral norm of weight matrices.
A Theoretical Analysis of Deep Q-Learning
Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood.
On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator
We study the global convergence of generative adversarial imitation learning for linear quadratic regulators, which is posed as minimax optimization.
Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning.
A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning
This paper considers a distributed reinforcement learning problem in which a network of multiple agents aim to cooperatively maximize the globally averaged return through communication with only local neighbors.
More Supervision, Less Computation: Statistical-Computational Tradeoffs in Weakly Supervised Learning
We consider the weakly supervised binary classification problem where the labels are randomly flipped with probability $1- {\alpha}$.
On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind.
Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization
We study the policy evaluation problem in multi-agent reinforcement learning where a group of agents, with jointly observed states and private local actions and rewards, collaborate to learn the value function of a given policy via local computation and communication over a connected undirected network.
Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
In detail, we prove that neural natural policy gradient converges to a globally optimal policy at a sublinear rate.
Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost.
Variance Reduced Policy Evaluation with Smooth Function Approximation
Policy evaluation with smooth and nonlinear function approximation has shown great potential for reinforcement learning.
Provably Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind.
Statistical-Computational Tradeoff in Single Index Models
Using the statistical query model to characterize the computational cost of an algorithm, we show that when $\cov(Y, X^\top\beta^*)=0$ and $\cov(Y,(X^\top\beta^*)^2)>0$, no computationally tractable algorithms can achieve the information-theoretic limit of the minimax risk.
Neural Trust Region/Proximal Policy Optimization Attains Globally Optimal Policy
Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning.
Neural Temporal-Difference Learning Converges to Global Optima
Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning.
Provably Efficient Exploration in Policy Optimization
While policy-based reinforcement learning (RL) achieves tremendous successes in practice, it is significantly less understood in theory, especially compared with value-based RL.
Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
Multi-agent reinforcement learning has been successfully applied to a number of challenging problems.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
On Computation and Generalization of Generative Adversarial Imitation Learning
Generative Adversarial Imitation Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies.
Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
In the offline setting, we control both players and aim to find the Nash Equilibrium by minimizing the duality gap.
Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
To this end, we present an \underline{O}ptimistic \underline{P}rimal-\underline{D}ual Proximal Policy \underline{OP}timization (OPDOP) algorithm where the value function is estimated by combining the least-squares policy evaluation and an additional bonus term for safe exploration.
Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Loss
In particular, we prove that the proposed algorithm achieves $\widetilde{\mathcal{O}}(L|\mathcal{S}|\sqrt{|\mathcal{A}|T})$ upper bounds of both the regret and the constraint violation, where $L$ is the length of each episode.
Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees
We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution.
Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate
Generative adversarial imitation learning (GAIL) demonstrates tremendous success in practice, especially when combined with neural networks.
Deep Reinforcement Learning with Robust and Smooth Policy
Deep reinforcement learning (RL) has achieved great empirical successes in various domains.
Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
We aim to answer the following questions: When the function approximator is a neural network, how does the associated feature representation evolve?
Neural Certificates for Safe Control Policies
This paper develops an approach to learn a policy of a dynamical system that is guaranteed to be both provably safe and goal-reaching.
Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
Empowered by expressive function approximators such as neural networks, deep reinforcement learning (DRL) achieves tremendous empirical successes.
Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning
We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
On the Global Optimality of Model-Agnostic Meta-Learning
Model-agnostic meta-learning (MAML) formulates meta-learning as a bilevel optimization problem, where the inner level solves each subtask based on a shared prior, while the outer level searches for the optimal shared prior by optimizing its aggregated performance over all the subtasks.
Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
We study risk-sensitive reinforcement learning in episodic Markov decision processes with unknown transition kernels, where the goal is to optimize the total reward under the risk measure of exponential utility.
Dynamic Regret of Policy Optimization in Non-stationary Environments
We consider reinforcement learning (RL) in episodic MDPs with adversarial full-information reward feedback and unknown fixed transition kernels.
Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
We study estimation in a class of generalized SEMs where the object of interest is defined as the solution to a linear operator equation.
Accelerating Nonconvex Learning via Replica Exchange Langevin Diffusion
To attain the advantages of both regimes, we propose to use replica exchange, which swaps between two Langevin diffusions with different temperatures.
A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic
Bilevel optimization is a class of problems which exhibit a two-level structure, and its goal is to minimize an outer objective function with variables which are constrained to be the optimal solution to an (inner) optimization problem.
Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
To the best of our knowledge, we establish the rate of convergence and global optimality of single-timescale actor-critic with linear function approximation for the first time.
Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment.
Single-Timescale Stochastic Nonconvex-Concave Optimization for Smooth Nonlinear TD Learning
Existing approaches for this problem are based on two-timescale or double-loop stochastic gradient algorithms, which may also require sampling large-batch data.
Generative Adversarial Imitation Learning with Neural Network Parameterization: Global Optimality and Convergence Rate
Generative adversarial imitation learning (GAIL) demonstrates tremendous success in practice, especially when combined with neural networks.
Deep Reinforcement Learning with Smooth Policy
In contrast to policy parameterized by linear/reproducing kernel functions, where simple regularization techniques suffice to control smoothness, for neural network based reinforcement learning algorithms, there is no readily available solution to learn a smooth policy.
Breaking the Curse of Many Agents: Provable Mean Embedding $Q$-Iteration for Mean-Field Reinforcement Learning
We highlight that MF-FQI algorithm enjoys a ``blessing of many agents'' property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Computational and Statistical Tradeoffs in Inferring Combinatorial Structures of Ising Model
We study the computational and statistical tradeoffs in inferring combinatorial structures of high dimensional simple zero-field ferromagnetic Ising model.
Nearly Dimension-Independent Sparse Linear Bandit over Small Action Spaces via Best Subset Selection
We consider the stochastic contextual bandit problem under the high dimensional linear model.
Policy Optimization in Zero-Sum Markov Games: Fictitious Self-Play Provably Attains Nash Equilibria
Specifically, in each iteration, each player infers the policy of the opponent implicitly via policy evaluation and improves its current policy by taking the smoothed best-response via a proximal policy optimization (PPO) step.
Offline Policy Optimization with Variance Regularization
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications.
Optimistic Policy Optimization with General Function Approximations
Although policy optimization with neural networks has a track record of achieving state-of-the-art results in reinforcement learning on various domains, the theoretical understanding of the computational and sample efficiency of policy optimization remains restricted to linear function approximations with finite-dimensional feature representations, which hinders the design of principled, effective, and efficient algorithms.
Entropic Risk-Sensitive Reinforcement Learning: A Meta Regret Framework with Function Approximation
We study risk-sensitive reinforcement learning with the entropic risk measure and function approximation.
Optimistic Exploration with Backward Bootstrapped Bonus for Deep Reinforcement Learning
However, such an approach is challenging in developing practical exploration algorithms for Deep Reinforcement Learning (DRL).
Provable Fictitious Play for General Mean-Field Games
We propose a reinforcement learning algorithm for stationary mean-field games, where the goal is to learn a pair of mean-field state and stationary policy that constitutes the Nash equilibrium.
End-to-End Learning and Intervention in Games
In this paper, we provide a unified framework for learning and intervention in games.
On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces
The classical theory of reinforcement learning (RL) has focused on tabular and linear representations of value functions.
Provably Efficient Neural Estimation of Structural Equation Models: An Adversarial Approach
We study estimation in a class of generalized SEMs where the object of interest is defined as the solution to a linear operator equation.
Provably Efficient Neural GTD for Off-Policy Learning
This paper studies a gradient temporal difference (GTD) algorithm using neural network (NN) function approximators to minimize the mean squared Bellman error (MSBE).
Provably Efficient Reinforcement Learning with Kernel and Neural Function Approximations
Reinforcement learning (RL) algorithms combined with modern function approximators such as kernel functions and deep neural networks have achieved significant empirical successes in large-scale application problems with a massive number of states.
Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
Temporal-difference and Q-learning play a key role in deep reinforcement learning, where they are empowered by expressive nonlinear function approximators such as neural networks.
Variational Transport: A Convergent Particle-BasedAlgorithm for Distributional Optimization
Specifically, we prove that moving along the geodesic in the direction of functional gradient with respect to the second-order Wasserstein distance is equivalent to applying a pushforward mapping to a probability distribution, which can be approximated accurately by pushing a set of particles.
Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
In particular, we focus on a variance-constrained policy optimization problem where the goal is to find a policy that maximizes the expected value of the long-run average reward, subject to a constraint that the long-run variance of the average reward is upper bounded by a threshold.
Is Pessimism Provably Efficient for Offline RL?
We study offline reinforcement learning (RL), which aims to learn an optimal policy based on a dataset collected a priori.
Provably Training Overparameterized Neural Network Classifiers with Non-convex Constraints
Training a classifier under non-convex constraints has gotten increasing attention in the machine learning community thanks to its wide range of applications such as algorithmic fairness and class-imbalanced classification.
A Primal-Dual Approach to Constrained Markov Decision Processes
In many operations management problems, we need to make decisions sequentially to minimize the cost while satisfying certain constraints.
Optimization and Control
A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentum
We focus on bilevel problems where the lower level subproblem is strongly-convex and the upper level objective function is smooth.
Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
Instrumental variables (IVs), in the context of RL, are the variables whose influence on the state variables are all mediated through the action.
Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
We also show that the overall convergence of DR-Off-PAC is doubly robust to the approximation errors that depend only on the expressive power of approximation functions.
Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
To exploit the permutation invariance therein, we propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture.
Verification in the Loop: Correct-by-Construction Control Learning with Reach-avoid Guarantees
Specifically, we leverage the verification results (computed reachable set of the system state) to construct feedback metrics for control learning, which measure how likely the current design of control parameters can meet the required reach-avoid property for safety and goal-reaching.
Gap-Dependent Bounds for Two-Player Markov Games
As one of the most popular methods in the field of reinforcement learning, Q-learning has received increasing attention.
A Unified Off-Policy Evaluation Approach for General Value Function
We further show that unlike GTD, the learned GVFs by GenTD are guaranteed to converge to the ground truth GVFs as long as the function approximation power is sufficiently large.
Towards General Function Approximation in Zero-Sum Markov Games
In the {coordinated} setting where both players are controlled by the agent, we propose a model-based algorithm and a model-free algorithm.
Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning
The recent emergence of reinforcement learning has created a demand for robust statistical inference methods for the parameter estimates computed using these algorithms.
Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
In generative adversarial imitation learning (GAIL), the agent aims to learn a policy from an expert demonstration so that its performance cannot be discriminated from the expert policy on a certain predefined reward set.
Inducing Equilibria via Incentives: Simultaneous Design-and-Play Ensures Global Convergence
To regulate a social system comprised of self-interested agents, economic incentives are often required to induce a desirable outcome.
A Principled Permutation Invariant Approach to Mean-Field Multi-Agent Reinforcement Learning
To exploit the permutation invariance therein, we propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation- invariant actor-critic neural architecture.
Can Reinforcement Learning Efficiently Find Stackelberg-Nash Equilibria in General-Sum Markov Games?
To our best knowledge, we establish the first provably efficient RL algorithms for solving SNE in general-sum Markov games with leader-controlled state transitions.
Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs
We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs).
On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Game
Then, given any extrinsic reward, the agent computes the policy via a planning algorithm with offline data collected in the exploration phase.
Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning
The exponential Bellman equation inspires us to develop a novel analysis of Bellman backup procedures in risk-sensitive RL algorithms, and further motivates the design of a novel exploration mechanism.
FinRL-Podracer: High Performance and Scalable Deep Reinforcement Learning for Quantitative Finance
Unfortunately, the steep learning curve and the difficulty in quick modeling and agile development are impeding finance researchers from using deep reinforcement learning in quantitative trading.
BooVI: Provably Efficient Bootstrapped Value Iteration
Despite the tremendous success of reinforcement learning (RL) with function approximation, efficient exploration remains a significant challenge, both practically and theoretically.
Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration
In constrained multi-objective RL, the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint.
Multi-Objective Reinforcement Learning
reinforcement-learning
Convergent Reinforcement Learning with Function Approximation: A Bilevel Optimization Perspective
We study reinforcement learning algorithms with nonlinear function approximation in the online setting.
Credible Sample Elicitation by Deep Learning, for Deep Learning
While classical elicitation results apply to eliciting a complex and generative (and continuous) distribution $p(x)$ for this image data, we are interested in eliciting samples $x_i \sim p(x)$ from agents.
Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?
We develop sample-efficient reinforcement learning (RL) algorithms for solving for an SNE in both online and offline settings.
Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic
Specifically, we consider a version of AC where the actor and critic are represented by overparameterized two-layer neural networks and are updated with two-timescale learning rates.
Joint Differentiable Optimization and Verification for Certified Reinforcement Learning
In model-based reinforcement learning for safety-critical control systems, it is important to formally certify system properties (e. g., safety, stability) under the learned controller.
Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasets
When the dataset does not have uniform coverage over all policy pairs, finding an approximate NE involves challenges in three aspects: (i) distributional shift between the behavior policy and the optimal policy, (ii) function approximation to handle large state space, and (iii) minimax optimization for equilibrium solving.
Sequential Information Design: Markov Persuasion Process and Its Efficient Reinforcement Learning
This paper proposes a novel model of sequential information design, namely the Markov persuasion processes (MPPs), where a sender, with informational advantage, seeks to persuade a stream of myopic receivers to take actions that maximizes the sender's cumulative utilities in a finite horizon Markovian environment with varying prior and utility functions.
Learning Dynamic Mechanisms in Unknown Environments: A Reinforcement Learning Approach
Dynamic mechanism design studies how mechanism designers should allocate resources among agents in a time-varying environment.
Learn to Match with No Regret: Reinforcement Learning in Markov Matching Markets
We study a Markov matching market involving a planner and a set of strategic agents on the two sides of the market.
Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiency
The sample efficiency of OP-TENET is enabled by a sequence of ingredients: (i) a Bellman operator with finite memory, which represents the value function in a recursive manner, (ii) the identification and estimation of such an operator via an adversarial integral equation, which features a smoothed discriminator tailored to the linear structure, and (iii) the exploration of the observation and state spaces via optimism, which is based on quantifying the uncertainty in the adversarial integral equation.
Pessimism meets VCG: Learning Dynamic Mechanism Design via Offline Reinforcement Learning
In the setting where the function approximation is employed to handle large state spaces, with only mild assumptions on the expressiveness of the function class, we are able to design a dynamic mechanism using offline reinforcement learning algorithms.
Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation
To the best of our knowledge, this is the first theoretical result for PbRL with (general) function approximation.
Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency
For a class of POMDPs with a low-rank structure in the transition kernel, ETC attains an $O(1/\epsilon^2)$ sample complexity that scales polynomially with the horizon and the intrinsic dimension (that is, the rank).
Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
We study offline reinforcement learning (RL) in partially observable Markov decision processes.
Federated Offline Reinforcement Learning
Extensive simulations demonstrate the effectiveness of the proposed algorithm.
Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitions
Our algorithms feature a combination of Upper Confidence Bound (UCB)-type optimism and fictitious play under the scope of simultaneous policy optimization in a non-stationary environment.
Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
Due to the lack of online interaction with the environment, offline RL is facing the following two significant challenges: (i) the agent may be confounded by the unobserved state variables; (ii) the offline data collected a prior does not provide sufficient coverage for the environment.
Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL
The cooperative Multi-A gent R einforcement Learning (MARL) with permutation invariant agents framework has achieved tremendous empirical successes in real-world applications.
Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments
It is quite challenging to ensure the safety of reinforcement learning (RL) agents in an unknown and stochastic environment under hard constraints that require the system state not to reach certain specified unsafe regions.
A Reinforcement Learning Approach in Multi-Phase Second-Price Auction Design
First, from the seller's perspective, we need to efficiently explore the environment in the presence of potentially nontruthful bidders who aim to manipulates seller's policy.
GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond
The proposed algorithm modifies the standard posterior sampling algorithm in two aspects: (i) we use an optimistic prior distribution that biases towards hypotheses with higher values and (ii) a loglikelihood function is set to be the empirical loss evaluated on the historical data, where the choice of loss function supports both model-free and model-based learning.
Latent Variable Representation for Reinforcement Learning
Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings.
Model-based Reinforcement Learning
reinforcement-learning
+1
Policy learning "without'' overlap: Pessimism and generalized empirical Bernstein's inequality
Existing policy learning methods rely on a uniform overlap assumption, i. e., the propensities of exploring all actions for all individual characteristics are lower bounded in the offline dataset.
Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information
To tackle the distributional mismatch, we leverage the idea of pessimism and use our OPE method to develop an off-policy learning algorithm for finding a desirable policy pair for both Alice and Bob.
Offline Policy Optimization in RL with Variance Regularizaton
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications.
An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models
In particular, such a representation instantiates the posterior distribution of the latent variable given input tokens, which plays a central role in predicting output labels and solving downstream tasks.
Differentiable Arbitrating in Zero-sum Markov Games
We initiate the study of how to perturb the reward in a zero-sum Markov game with two players to induce a desirable Nash equilibrium, namely arbitrating.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learning
We study a heterogeneous agent macroeconomic model with an infinite number of households and firms competing in a labor market.
A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observations
We study the offline contextual bandit problem, where we aim to acquire an optimal policy using observational data.
Wardrop Equilibrium Can Be Boundedly Rational: A New Behavioral Theory of Route Choice
We achieve this result by developing a day-to-day (DTD) dynamical model that mimics how travelers gradually adjust their route valuations, hence choice probabilities, based on past experiences.
What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization
(b) What is a proper performance metric for ICL and what is the error rate?
Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning
We examine online safe multi-agent reinforcement learning using constrained Markov games in which agents compete by maximizing their expected total rewards under a constraint on expected total utilities.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
A General Framework for Sequential Decision-Making under Adaptivity Constraints
We take the first step in studying general sequential decision-making under two adaptivity constraints: rare policy switch and batch learning.
Contextual Dynamic Pricing with Strategic Buyers
We first prove that existing non-strategic pricing policies that neglect the buyers' strategic behavior result in a linear $\Omega(T)$ regret with $T$ the total time horizon, indicating that these policies are not better than a random pricing policy.
Let Models Speak Ciphers: Multiagent Debate through Embeddings
Although natural language is an obvious choice for communication due to LLM's language understanding capability, the token sampling step needed when generating natural language poses a potential risk of information loss, as it uses only one token to represent the model's belief across the entire vocabulary.
Sample-Efficient Multi-Agent RL: An Optimization Perspective
We study multi-agent reinforcement learning (MARL) for the general-sum Markov Games (MGs) under the general function approximation.
Learning Regularized Graphon Mean-Field Games with Unknown Graphons
Second, using kernel embedding of distributions, we design efficient algorithms to estimate the transition kernels, reward functions, and graphons from sampled agents.
A Principled Framework for Knowledge-enhanced Large Language Model
Large Language Models (LLMs) are versatile, yet they often falter in tasks requiring deep and reliable reasoning due to issues like hallucinations, limiting their applicability in critical scenarios.
Provably Efficient High-Dimensional Bandit Learning with Batched Feedbacks
For these settings, we design a provably sample-efficient algorithm which achieves a $ \mathcal{\tilde O}(s_0^2 \log^2 T)$ regret in the sparse case and $ \mathcal{\tilde O} ( r ^2 \log^2 T)$ regret in the low-rank case, using only $L = \mathcal{O}( \log T)$ batches.
Empowering Autonomous Driving with Large Language Models: A Safety Perspective
Autonomous Driving (AD) encounters significant safety hurdles in long-tail unforeseen driving scenarios, largely stemming from the non-interpretability and poor generalization of the deep neural networks within the AD system, particularly in out-of-distribution and uncertain data.
Sparse PCA with Oracle Property
In particular, under a weak assumption on the magnitude of the population projection matrix, one estimator within this family exactly recovers the true support with high probability, has exact rank-$k$, and attains a $\sqrt{s/n}$ statistical rate of convergence with $s$ being the subspace sparsity level and $n$ the sample size.
Human-Instruction-Free LLM Self-Alignment with Limited Samples
The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples.
Double Duality: Variational Primal-Dual Policy Optimization for Constrained Reinforcement Learning
Designing algorithms for a constrained convex MDP faces several challenges, including (1) handling the large state space, (2) managing the exploration/exploitation tradeoff, and (3) solving the constrained optimization where the objective and the constraint are both nonlinear functions of the visitation measure.
How Can LLM Guide RL? A Value-Based Approach
Specifically, we develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning, particularly when the difference between the ideal policy and the LLM-informed policy is small, which suggests that the initial policy is close to optimal, reducing the need for further exploration.
Can Large Language Models Play Games? A Case Study of A Self-Play Approach
Large Language Models (LLMs) harness extensive data from the Internet, storing a broad spectrum of prior knowledge.
A Mean-Field Analysis of Neural Gradient Descent-Ascent: Applications to Functional Conditional Moment Equations
Under this regime, gradient descent-ascent corresponds to a Wasserstein gradient flow over the space of probability measures defined over the space of neural network parameters.
