Search Results for author:
Found 99 papers, 23 papers with code
An Optimistic Perspective on Offline Deep Reinforcement Learning
The DQN replay dataset can serve as an offline RL benchmark and is open-sourced.
Foundation Models for Decision Making: Problems, Methods, and Opportunities
In response to these developments, new paradigms are emerging for training foundation models to interact with other agents and perform long-term reasoning.
Gradient-Free Structured Pruning with Unlabeled Data
By only using the weights of the pre-trained model and unlabeled data, in a matter of a few minutes on a single GPU, up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.
Learning Universal Policies via Text-Guided Video Generation
The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks.
The Role of Baselines in Policy Gradient Optimization
Instead, the analysis reveals that the primary effect of the value baseline is to \textbf{reduce the aggressiveness of the updates} rather than their variance.
Memory Augmented Large Language Models are Computationally Universal
We show that transformer-based large language models are computationally universal when augmented with an external memory.
Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off
Importantly, these two errors behave differently with respect to time discretization, which implies that there is an optimal choice for the temporal resolution that depends on the data budget.
Latent Variable Representation for Reinforcement Learning
Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings.
Model-based Reinforcement Learning
reinforcement-learning
+1
A Simple Decentralized Cross-Entropy Method
To tackle this issue, we propose Decentralized CEM (DecentCEM), a simple but effective improvement over classical CEM, by using an ensemble of CEM instances running independently from one another, and each performing a local improvement of its own sampling distribution.
Score-based Continuous-time Discrete Diffusion Models
Score-based modeling through stochastic differential equations (SDEs) has provided a new perspective on diffusion models, and demonstrated superior performance on continuous data.
What learning algorithm is in-context learning? Investigations with linear models
We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context.
TEMPERA: Test-Time Prompting via Reinforcement Learning
To achieve this, we design a novel action space that allows flexible editing of the initial prompts covering a wide set of commonly-used components like instructions, few-shot exemplars, and verbalizers.
Learning to Optimize with Stochastic Dominance Constraints
In real-world decision-making, uncertainty is important yet difficult to handle.
Dichotomy of Control: Separating What You Can Control from What You Cannot
While return-conditioning is at the heart of popular algorithms such as decision transformer (DT), these methods tend to perform poorly in highly stochastic environments, where an occasional high return can arise from randomness in the environment rather than the actions themselves.
Optimal Scaling for Locally Balanced Proposals in Discrete Spaces
Optimal scaling has been well studied for Metropolis-Hastings (M-H) algorithms in continuous spaces, but a similar understanding has been lacking in discrete spaces.
Spectral Decomposition Representation for Reinforcement Learning
Representation learning often plays a critical role in reinforcement learning by managing the curse of dimensionality.
Making Linear MDPs Practical via Contrastive Representation Learning
It is common to address the curse of dimensionality in Markov decision processes (MDPs) by exploiting low-rank representations.
Rationale-Augmented Ensembles in Language Models
Recent research has shown that rationales, or step-by-step chains of thought, can be used to improve performance in multi-step reasoning tasks.
Discrete Langevin Sampler via Wasserstein Gradient Flow
It is known that gradient-based MCMC samplers for continuous spaces, such as Langevin Monte Carlo (LMC), can be derived as particle versions of a gradient flow that minimizes KL divergence on a Wasserstein manifold.
A Parametric Class of Approximate Gradient Updates for Policy Optimization
Approaches to policy optimization have been motivated from diverse principles, based on how the parametric model is interpreted (e. g. value versus policy representation) or how the learning objective is formulated, yet they share a common goal of maximizing expected return.
Multimodal Masked Autoencoders Learn Transferable Representations
We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks.
Chain of Thought Imitation with Procedure Cloning
Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior.
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Although chain-of-thought prompting has shown impressive results on many natural language reasoning tasks, it often performs poorly on tasks which need to solve problems harder than the demonstration examples.
Ranked #14 on
Arithmetic Reasoning
on GSM8K
Reinforcement Teaching
To effectively learn such a teaching policy, we introduce a parametric-behavior embedder that learns a representation of the student's learnable parameters from its input/output behavior.
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks.
Ranked #9 on
Arithmetic Reasoning
on GSM8K
(using extra training data)
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning.
Neural Stochastic Dual Dynamic Programming
Stochastic dual dynamic programming (SDDP) is a state-of-the-art method for solving multi-stage stochastic optimization, widely used for modeling real-world process optimization tasks.
Understanding the Effect of Stochasticity in Policy Optimization
We study the effect of stochasticity in on-policy policy optimization, and make the following four contributions.
SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs
There are two important reasoning tasks on KGs: (1) single-hop knowledge graph completion, which involves predicting individual links in the KG; and (2), multi-hop reasoning, where the goal is to predict which KG entities satisfy a given logical query.
Decentralized Cross-Entropy Method for Model-Based Reinforcement Learning
Further, we extend the decentralized approach to sequential decision-making problems where we show in 13 continuous control benchmark environments that it matches or outperforms the state-of-the-art CEM algorithms in most cases, under the same budget of the total number of samples for planning.
Disentangling Generalization in Reinforcement Learning
The resulting environment, while simple, necessitates function approximation for state abstraction and provides ground-truth labels for optimal policies and value functions.
Understanding and Leveraging Overparameterization in Recursive Value Estimation
To better understand the utility of deep models in RL we present an analysis of recursive value estimation using overparameterized linear representations that provides useful, transferable findings.
Combiner: Full Attention Transformer with Sparse Computation Cost
However, the key limitation of transformers is their quadratic memory and time complexity $\mathcal{O}(L^2)$ with respect to the sequence length in attention layers, which restricts application in extremely long sequences.
Ranked #2 on
Language Modelling
on Wiki-40B
The Curse of Passive Data Collection in Batch Reinforcement Learning
In high stake applications, active experimentation may be considered too risky and thus data are often collected passively.
Characterizing the Gap Between Actor-Critic and Policy Gradient
Actor-critic (AC) methods are ubiquitous in reinforcement learning.
Leveraging Non-uniformity in First-order Non-convex Optimization
Classical global convergence results for first-order methods rely on uniform smoothness and the \L{}ojasiewicz inequality.
Joint Attention for Multi-Agent Coordination and Social Learning
We then train agents to minimize the difference between the attention weights that they apply to the environment at each timestep, and the attention of other agents.
On the Optimality of Batch Policy Optimization Algorithms
First, we introduce a class of confidence-adjusted index algorithms that unifies optimistic and pessimistic principles in a common framework, which enables a general analysis.
Optimization Issues in KL-Constrained Approximate Policy Iteration
We compare the use of KL divergence as a constraint vs. as a regularizer, and point out several optimization issues with the widely-used constrained approach.
Offline Policy Selection under Uncertainty
More importantly, we show how the belief distribution estimated by BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric, and we empirically demonstrate that this selection procedure significantly outperforms existing approaches, such as ranking policies according to mean or high-confidence lower bound value estimates.
Escaping the Gravitational Pull of Softmax
Both findings are based on an analysis of convergence rates using the Non-uniform \L{}ojasiewicz (N\L{}) inequalities.
Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
In this paper we propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data, where parameter gradients are estimated using a learned sampler that mimics local search.
CoinDICE: Off-Policy Confidence Interval Estimation
We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies.
Attention that does not Explain Away
Models based on the Transformer architecture have achieved better accuracy than the ones based on competing architectures for a large set of tasks.
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
In this work, we closely investigate an important simplification of BCQ -- a prior approach for offline RL -- which removes a heuristic design choice and naturally restricts extracted policies to remain exactly within the support of a given behavior policy.
Off-Policy Evaluation via the Regularized Lagrangian
The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data.
Go Wide, Then Narrow: Efficient Training of Deep Thin Networks
This is achieved by layerwise imitation, that is, forcing the thin network to mimic the intermediate outputs of the wide network from layer to layer.
Scalable Deep Generative Modeling for Sparse Graphs
Based on this, we develop a novel autoregressive model, named BiGG, that utilizes this sparsity to avoid generating the full adjacency matrix, and importantly reduces the graph generation time complexity to $O((n + m)\log n)$.
A maximum-entropy approach to off-policy evaluation in average-reward MDPs
This work focuses on off-policy evaluation (OPE) with function approximation in infinite-horizon undiscounted Markov decision processes (MDPs).
On the Global Convergence Rates of Softmax Policy Gradient Methods
First, we show that with the true gradient, policy gradient with a softmax parametrization converges at a $O(1/t)$ rate, with constants depending on the problem and initialization.
Energy-Based Processes for Exchangeable Data
Recently there has been growing interest in modeling sets with exchangeability such as point clouds.
Variational Inference for Deep Probabilistic Canonical Correlation Analysis
In this paper, we propose a deep probabilistic multi-view model that is composed of a linear multi-view layer based on probabilistic canonical correlation analysis (CCA) description in the latent space together with deep generative networks as observation models.
Batch Stationary Distribution Estimation
We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions.
ConQUR: Mitigating Delusional Bias in Deep Q-learning
Delusional bias is a fundamental source of error in approximate Q-learning.
GenDICE: Generalized Offline Estimation of Stationary Values
An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain.
Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate.
Model-based Reinforcement Learning
reinforcement-learning
+1
AlgaeDICE: Policy Gradient from Arbitrary Experience
In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility.
Surrogate Objectives for Batch Policy Optimization in One-step Decision Making
We investigate batch policy optimization for cost-sensitive classification and contextual bandits---two related tasks that obviate exploration but require generalizing from observed rewards to action selections in unseen contexts.
Maximum Entropy Monte-Carlo Planning
We then extend this approach to general sequential decision making by developing a general MCTS algorithm, Maximum Entropy for Tree Search (MENTS).
Invertible Convolutional Flow
We show that these transforms allow more effective normalizing flow models to be developed for generative image models.
Striving for Simplicity in Off-Policy Deep Reinforcement Learning
This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms.
Domain Aggregation Networks for Multi-Source Domain Adaptation
In many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target dataset -- e. g., recognizing characters of a new font using a set of different fonts.
An Optimistic Perspective on Offline Reinforcement Learning
The DQN replay dataset can serve as an offline RL benchmark and is open-sourced.
Advantage Amplification in Slowly Evolving Latent-State Environments
Latent-state environments with long horizons, such as those faced by recommender systems, pose significant challenges for reinforcement learning (RL).
Exponential Family Estimation via Adversarial Dynamics Embedding
We present an efficient algorithm for maximum likelihood estimation (MLE) of exponential family models, with a general parametrization of the energy function that includes neural networks.
Learning to Generalize from Sparse and Underspecified Rewards
The parameters of the auxiliary reward function are optimized with respect to the validation performance of a trained policy.
The Value Function Polytope in Reinforcement Learning
We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes.
A Geometric Perspective on Optimal Representations for Reinforcement Learning
We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks.
Non-delusional Q-learning and value-iteration
We identify a fundamental source of error in Q-learning and other forms of dynamic programming with function approximation.
Understanding the impact of entropy on policy optimization
Entropy regularization is commonly used to improve policy optimization in reinforcement learning.
Kernel Exponential Family Estimation via Doubly Dual Embedding
We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space.
Planning and Learning with Stochastic Action Sets
From an RL perspective, we show that Q-learning with sampled action sets is sound.
Variational Rejection Sampling
Learning latent variable models with stochastic variational inference is challenging when the approximate posterior is far from the true posterior, due to high variance in the gradient estimates.
Smoothed Action Value Functions for Learning Gaussian Policies
State-action value functions (i. e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning.
Learning Gaussian Policies from Smoothed Action Value Functions
We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value used in SARSA.
Multi-view Matrix Factorization for Linear Dynamical System Estimation
In this paper, we instead reconsider likelihood maximization and develop an optimization based strategy for recovering the latent states and transition parameters.
Safe Exploration for Identifying Linear Systems via Robust Optimization
Motivated by the cooling of Google's data centers, we study how one can safely identify the parameters of a system model with a desired accuracy and confidence level.
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
When evaluated on a number of continuous control tasks, Trust-PCL improves the solution quality and sample efficiency of TRPO.
Bridging the Gap Between Value and Policy Based Reinforcement Learning
We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization.
Deep Learning Games
We investigate a reduction of supervised learning to game playing that reveals new connections and learning methods.
Improving Policy Gradient by Exploring Under-appreciated Rewards
We propose a more directed exploration strategy that promotes exploration of under-appreciated reward regions.
Reward Augmented Maximum Likelihood for Neural Structured Prediction
A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation.
Stochastic Neural Networks with Monotonic Activation Functions
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise.
Embedding Inference for Structured Multilabel Prediction
A key bottleneck in structured output prediction is the need for inference during training and testing, usually requiring some form of dynamic programming.
Semi-Supervised Zero-Shot Classification With Label Representation Learning
Most existing zero-shot learning methods require a user to first provide a set of semantic visual attributes for each class as side information before applying a two-step prediction procedure that introduces an intermediate attribute prediction problem.
Learning with a Strong Adversary
The robustness of neural networks to intended perturbations has recently attracted significant attention.
Convex Deep Learning via Normalized Kernels
Deep learning has been a long standing pursuit in machine learning, which until recently was hampered by unreliable training methods before the discovery of improved heuristics for embedded layer training.
Generalized Conditional Gradient for Sparse Estimation
Structured sparsity is an important modeling tool that expands the applicability of convex formulations for data analysis, however it also creates significant challenges for efficient algorithm design.
Adaptive Monte Carlo via Bandit Allocation
We consider the problem of sequentially choosing between a set of unbiased Monte Carlo estimators to minimize the mean-squared-error (MSE) of a final combined estimate.
Polar Operators for Structured Sparse Estimation
Structured sparse estimation has become an important technique in many areas of data analysis.
Convex Two-Layer Modeling
Latent variable prediction models, such as multi-layer networks, impose auxiliary latent variables between inputs and outputs to allow automatic inference of implicit features useful for prediction.
Convex Relaxations of Bregman Divergence Clustering
Although many convex relaxations of clustering have been proposed in the past decade, current formulations remain restricted to spherical Gaussian or discriminative models and are susceptible to imbalanced clusters.
Accelerated Training for Matrix-norm Regularization: A Boosting Approach
Sparse learning models typically combine a smooth loss with a nonsmooth penalty, such as trace norm.
Convex Multi-view Subspace Learning
Subspace learning seeks a low dimensional representation of data that enables accurate reconstruction.
A Polynomial-time Form of Robust Regression
Despite the variety of robust regression methods that have been developed, current regression formulations are either NP-hard, or allow unbounded response to even a single leverage point.
Relaxed Clipping: A Global Training Method for Robust Regression and Classification
We present a generic procedure that can be applied to standard loss functions and demonstrate improved robustness in regression and classification problems.
Convex Relaxation of Mixture Regression with Efficient Algorithms
The second is a min-min reformulation consisting of fast alternating steps of closed-form updates.
A General Projection Property for Distribution Families
We prove that linear projections between distribution families with fixed first and second moments are surjective, regardless of dimension.
Discriminative Batch Mode Active Learning
Most previous studies in active learning have focused on selecting one unlabeled instance at one time while retraining in each iteration.
