Search Results for author:
Found 13 papers, 3 papers with code
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
Such approaches suffer from covariate shift: the conversations in the training set have previous turns generated by some reference policy, which means that low training error may not necessarily correspond to good performance when the learner is actually in the conversation loop.
Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank
We study the problem of learning an approximate equilibrium in the offline multi-agent reinforcement learning (MARL) setting.
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
Language model alignment methods, such as reinforcement learning from human feedback (RLHF), have led to impressive advances in language model capabilities, but existing techniques are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model plateaus or degrades over the course of the alignment process.
REBEL: Reinforcement Learning via Regressing Relative Rewards
While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications, including the fine-tuning of generative models.
Dataset Reset Policy Optimization for RLHF
Motivated by the fact that offline preference dataset provides informative states (i. e., data that is preferred by the labelers), our new algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing offline preference dataset into the online policy training procedure via dataset reset: it directly resets the policy optimizer to the states in the offline dataset, instead of always starting from the initial state distribution.
Optimal Multi-Distribution Learning
Focusing on a hypothesis class of Vapnik-Chervonenkis (VC) dimension d, we propose a novel algorithm that yields an varepsilon-optimal randomized hypothesis with a sample complexity on the order of (d+k)/varepsilon^2 (modulo some logarithmic factor), matching the best-known lower bound.
Provably Efficient CVaR RL in Low-rank MDPs
We study CVaR RL in low-rank MDPs with nonlinear function approximation.
Provable Reward-Agnostic Preference-Based Reinforcement Learning
Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than explicit reward signals.
Provable Offline Preference-Based Reinforcement Learning
Our proposed algorithm consists of two main steps: (1) estimate the implicit reward using Maximum Likelihood Estimation (MLE) with general function approximation from offline data and (2) solve a distributionally robust planning problem over a confidence set around the MLE.
PAC Reinforcement Learning for Predictive State Representations
We show that given a realizable model class, the sample complexity of learning the near optimal policy only scales polynomially with respect to the statistical complexity of the model class, without any explicit polynomial dependence on the size of the state and observation spaces.
Decentralized Optimistic Hyperpolicy Mirror Descent: Provably No-Regret Learning in Markov Games
We study decentralized policy learning in Markov games where we control a single agent to play with nonstationary and possibly adversarial opponents.
Offline Reinforcement Learning with Realizability and Single-policy Concentrability
Sample-efficiency guarantees for offline reinforcement learning (RL) often rely on strong assumptions on both the function classes (e. g., Bellman-completeness) and the data coverage (e. g., all-policy concentrability).
Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
These can often be accounted for via regularized RL, which augments the target value function with a structure-promoting regularizer.
