Search Results for author:
Found 60 papers, 8 papers with code
Optimal policy evaluation using kernel-based temporal difference methods
Whereas existing worst-case theory predicts cubic scaling ($H^3$) in the effective horizon, our theory reveals that there is in fact a much wider range of scalings, depending on the kernel, the stationary distribution, and the variance of the Bellman residual error.
Boosting the Convergence of Reinforcement Learning-based Auto-pruning Using Historical Data
Reinforcement learning (RL)-based auto-pruning has been further proposed to automate the DNN pruning process to avoid expensive hand-crafted work.
On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control
To close this gap, we step towards persistent exploration in continuous space through policy parameterizations defined by distributions of heavier tails defined by tail-index parameter alpha, which increases the likelihood of jumping in state space.
You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient
Motivated by the necessity of efficient inference across various constraints on BERT, we propose a novel approach, YOCO-BERT, to achieve compress once and deploy everywhere.
1$\times$N Block Pattern for Network Sparsity
In particular, consecutive $N$ output kernels with the same input channel index are grouped into one block, which serves as a basic pruning granularity of our pruning pattern.
MARL with General Utilities via Decentralized Shadow Reward Actor-Critic
DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i. e., the "shadow reward".
Towards Compact CNNs via Collaborative Compression
Channel pruning and tensor decomposition have received extensive attention in convolutional neural network compression.
Learning Good State and Action Representations via Tensor Decomposition
The transition kernel of a continuous-state-action Markov decision process (MDP) admits a natural tensor structure.
On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
By assuming the overparameterizaiton of policy and exploiting the hidden convexity of the problem, we further show that TSIVR-PG converges to global $\epsilon$-optimal policy with $\tilde{\mathcal{O}}(\epsilon^{-2})$ samples.
Bootstrapping Statistical Inference for Off-Policy Evaluation
Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood.
Provably Efficient Reinforcement Learning with Kernel and Neural Function Approximations
Reinforcement learning (RL) algorithms combined with modern function approximators such as kernel functions and deep neural networks have achieved significant empirical successes in large-scale application problems with a massive number of states.
On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces
The classical theory of reinforcement learning (RL) has focused on tabular and linear representations of value functions.
Online Sparse Reinforcement Learning
First, we provide a lower bound showing that linear regret is generally unavoidable in this case, even if there exists a policy that collects well-conditioned data.
Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
To evaluate a new target policy, we analyze a Lasso fitted Q-evaluation method and establish a finite-sample error bound that has no polynomial dependence on the ambient dimension.
High-Dimensional Sparse Linear Bandits
Stochastic linear bandits with high-dimensional sparse features are a practical model for a variety of domains, including personalized medicine and online advertising.
Generalized Leverage Score Sampling for Neural Networks
Leverage score sampling is a powerful technique that originates from theoretical computer science, which can be used to speed up a large number of fundamental questions, e. g. linear regression, linear programming, semi-definite programming, cutting plane method, graph sparsification, maximum matching and max-flow.
Variational Policy Gradient Method for Reinforcement Learning with General Utilities
Analogously to the Policy Gradient Theorem \cite{sutton2000policy} available for RL with cumulative rewards, we derive a new Variational Policy Gradient Theorem for RL with general utilities, which establishes that the parametrized policy gradient may be obtained as the solution of a stochastic saddle point problem involving the Fenchel dual of the utility function.
Picasso: A Sparse Learning Library for High Dimensional Data Analysis in R and Python
We describe a new library named picasso, which implements a unified framework of pathwise coordinate optimization for a variety of sparse learning problems (e. g., sparse linear regression, sparse logistic regression, sparse Poisson regression and scaled sparse linear regression) combined with efficient active set selection strategies.
Model-Based Reinforcement Learning with Value-Targeted Regression
We propose a model based RL algorithm that is based on optimism principle: In each episode, the set of models that are `consistent' with the data collected is constructed.
Concept Annotation for Intelligent Textbooks
The outcomes of our work include a validated knowledge engineering procedure, a code-book for technical concept annotation, and a set of concept annotations for the target textbook, which could be used as gold standard in further research.
Cautious Reinforcement Learning via Distributional Risk in the Dual Domain
To ameliorate this issue, we propose a new definition of risk, which we call caution, as a penalty function added to the dual objective of the linear programming (LP) formulation of reinforcement learning.
Sketching Transformed Matrices with Applications to Natural Language Processing
We show that our approach obtains small error and is efficient in both space and time.
Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation
We prove that this method is information-theoretically optimal and has nearly minimal estimation error.
Contrastive Multi-document Question Generation
In the SL stage, a single-document question generator is trained.
Continuous Control with Contexts, Provably
To our knowledge, this is first provably efficient algorithm to build a decoder in the continuous control setting.
Characterizing Deep Learning Training Workloads on Alibaba-PAI
One critical issue for efficiently operating practical AI clouds, is to characterize the computing and data transfer demands of these workloads, and more importantly, the training performance given the underlying software framework and hardware configurations.
Solving Discounted Stochastic Two-Player Games with Near-Optimal Time and Sample Complexity
In this paper, we settle the sampling complexity of solving discounted two-player turn-based zero-sum stochastic games up to polylogarithmic factors.
Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT
The recent success of single-agent reinforcement learning (RL) in Internet of things (IoT) systems motivates the study of multi-agent reinforcement learning (MARL), which is more challenging but more useful in large-scale IoT.
Learning Markov models via low-rank optimization
We show that one can estimate the full transition model accurately using a trajectory of length that is proportional to the total number of states.
Feature-Based Q-Learning for Two-Player Stochastic Games
Consider a two-player zero-sum stochastic game where the transition function can be embedded in a given feature space.
Learning low-dimensional state embeddings and metastable clusters from time series data
This paper studies how to find compact state embeddings from high-dimensional Markov state trajectories, where the transition kernel has a small intrinsic rank.
RL4health: Crowdsourcing Reinforcement Learning for Knee Replacement Pathway Optimization
Joint replacement is the most common inpatient surgical treatment in the US.
Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
In this case, the kernelized MatrixRL satisfies a regret bound ${O}\big(H^2\widetilde{d}\log T\sqrt{T}\big)$, where $\widetilde{d}$ is the effective dimension of the kernel space.
Learning to Control in Metric Space with Optimal Regret
We study online reinforcement learning for finite-horizon deterministic control systems with {\it arbitrary} state and action spaces.
Sample-Optimal Parametric Q-Learning Using Linearly Additive Features
Consider a Markov decision process (MDP) that admits a set of state-action features, which can linearly express the process's probabilistic transition model.
Near-Optimal Time and Sample Complexities for Solving Markov Decision Processes with a Generative Model
In this paper we consider the problem of computing an $\epsilon$-optimal policy of a discounted Markov Decision Process (DMDP) provided we can only access its transition function through a generative sampling model that given any state-action pair samples from the transition function in $O(1)$ time.
Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks
In this method, the network is viewed as a computational graph, in which the vertices denote the computation nodes and edges represent the information flow.
State Aggregation Learning from Markov Transition Data
Our proposed method is a simple two-step algorithm: The first step is spectral decomposition of empirical transition matrix, and the second step conducts a linear transformation of singular vectors to find their approximate convex hull.
Towards Coherent and Cohesive Long-form Text Generation
Generating coherent and cohesive long-form texts is a challenging task.
Adaptive Low-Nonnegative-Rank Approximation for State Aggregation of Markov Chains
The efficiency and statistical properties of our approach are illustrated on synthetic data.
Diffusion Approximations for Online Principal Component Estimation and Global Convergence
In this paper, we propose to adopt the diffusion approximation tools to study the dynamics of Oja's iteration which is an online stochastic gradient descent method for the principal component analysis.
Scalable Bilinear Pi Learning Using State and Action Features
In this work, we study a primal-dual formulation of the ALP, and develop a scalable, model-free algorithm called bilinear $\pi$ learning for reinforcement learning when a sampling oracle is provided.
Near-Optimal Time and Sample Complexities for Solving Discounted Markov Decision Process with a Generative Model
In this paper we consider the problem of computing an $\epsilon$-optimal policy of a discounted Markov Decision Process (DMDP) provided we can only access its transition function through a generative sampling model that given any state-action pair samples from the transition function in $O(1)$ time.
Optimization and Control
Improved Sample Complexity for Stochastic Compositional Variance Reduced Gradient
Convex composition optimization is an emerging topic that covers a wide range of applications arising from stochastic optimal control, reinforcement learning and multi-stage stochastic programming.
Scalable Bilinear $π$ Learning Using State and Action Features
In this work, we study a primal-dual formulation of the ALP, and develop a scalable, model-free algorithm called bilinear $\pi$ learning for reinforcement learning when a sampling oracle is provided.
Estimation of Markov Chain via Rank-Constrained Likelihood
This paper studies the estimation of low-rank Markov chains from empirical trajectories.
Dimensionality Reduction for Stationary Time Series via Stochastic Nonconvex Optimization
Specifically, our goal is to estimate the principle component of time series data with respect to the covariance matrix of the stationary distribution.
Variance Reduction Methods for Sublinear Reinforcement Learning
There is a technical issue in the analysis that is not easily fixable.
Spectral State Compression of Markov Processes
Model reduction of Markov processes is a basic problem in modeling state-transition systems.
Improved Oracle Complexity of Variance Reduced Methods for Nonsmooth Convex Stochastic Composition Optimization
We consider the nonsmooth convex composition optimization problem where the objective is a composition of two finite-sum functions and analyze stochastic compositional variance reduced gradient (SCVRG) methods for them.
Deep Primal-Dual Reinforcement Learning: Accelerating Actor-Critic using Bellman Duality
We believe that the primal-dual updates to the value and policy functions would expedite the learning process.
Variance Reduced Value Iteration and Faster Algorithms for Solving Markov Decision Processes
Given a discounted Markov Decision Process (DMDP) with $|S|$ states, $|A|$ actions, discount factor $\gamma\in(0, 1)$, and rewards in the range $[-M, M]$, we show how to compute an $\epsilon$-optimal policy, with probability $1 - \delta$ in time \[ \tilde{O}\left( \left(|S|^2 |A| + \frac{|S| |A|}{(1 - \gamma)^3} \right) \log\left( \frac{M}{\epsilon} \right) \log\left( \frac{1}{\delta} \right) \right) ~ .
Primal-Dual $π$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems
Consider the problem of approximating the optimal policy of a Markov decision process (MDP) by sampling state transitions.
Online Factorization and Partition of Complex Networks From Random Walks
We formulate this into a nonconvex stochastic factorization problem and propose an efficient and scalable stochastic generalized Hebbian algorithm.
Lower Bound On the Computational Complexity of Discounted Markov Decision Problems
We study the computational complexity of the infinite-horizon discounted-reward Markov Decision Problem (MDP) with a finite state space $|\mathcal{S}|$ and a finite action space $|\mathcal{A}|$.
Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
We study the online estimation of the optimal policy of a Markov decision process (MDP).
Accelerating Stochastic Composition Optimization
The ASC-PG is the first proximal gradient method for the stochastic composition problem that can deal with nonsmooth regularization penalty.
Near-Optimal Stochastic Approximation for Online Principal Component Estimation
We prove for the first time a nearly optimal finite-sample error bound for the online PCA algorithm.
Random Multi-Constraint Projection: Stochastic Gradient Methods for Convex Optimization with Many Constraints
Consider convex optimization problems subject to a large number of constraints.
Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functions
For smooth convex problems, the SCGD can be accelerated to converge at a rate of $O(k^{-2/7})$ in the general case and $O(k^{-4/5})$ in the strongly convex case.