Search Results for author:
Found 21 papers, 4 papers with code
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i. e., employ a "negative gradient") outperform offline and maximum likelihood objectives.
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
In this paper, we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences.
Towards Principled Representation Learning from Videos for Reinforcement Learning
We study two types of settings: one where there is iid noise in the observation, and a more challenging setting where there is also the presence of exogenous noise, which is non-iid noise that is temporally correlated, such as the motion of people or cars in the background.
CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples
We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning.
Harnessing Density Ratios for Online Reinforcement Learning
The theories of offline and online reinforcement learning, despite having evolved in parallel, have begun to show signs of the possibility for a unification, with algorithms and analysis techniques for one setting often having natural counterparts in the other.
Adversarial Model for Offline Reinforcement Learning
We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage.
ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data
We propose a new model-based offline RL framework, called Adversarial Models for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary baseline policy regardless of data coverage.
The Role of Coverage in Online Reinforcement Learning
Coverage conditions -- which assert that the data logging distribution adequately covers the state space -- play a fundamental role in determining the sample complexity of offline reinforcement learning.
Interaction-Grounded Learning with Action-inclusive Feedback
Consider the problem setting of Interaction-Grounded Learning (IGL), in which a learner's goal is to optimally interact with the environment with no explicit reward to ground its policies.
Adversarially Trained Actor Critic for Offline Reinforcement Learning
We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism.
Bellman-consistent Pessimism for Offline Reinforcement Learning
The use of pessimism, when reasoning about datasets lacking exhaustive exploration has recently gained prominence in offline reinforcement learning.
Interaction-Grounded Learning
We propose Interaction-Grounded Learning for this novel setting, in which a learner's goal is to interact with the environment with no grounding or explicit reward to optimize its policies.
Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
This offline result is the first that matches the sample complexity lower bound in this setting, and resolves a recent open question in offline RL.
Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning using function approximation for marginal importance weights and $q$-functions when these are estimated using recent minimax methods.
A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Setting
Recently, Wang et al. (2020) showed a highly intriguing hardness result for batch reinforcement learning (RL) with linearly realizable value function and good feature coverage in the finite-horizon case.
Batch Value-function Approximation with Only Realizability
We make progress in a long-standing problem of batch reinforcement learning (RL): learning $Q^\star$ from an exploratory and polynomial-sized dataset, using a realizable and otherwise arbitrary function class.
Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison
We prove performance guarantees of two algorithms for approximating $Q^\star$ in batch reinforcement learning.
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
To solve this problem, we consider a marginalized importance sampling (MIS) estimator that recursively estimates the state marginal distribution for the target policy at every step.
Provably Efficient Q-Learning with Low Switching Cost
We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization.
Privacy Preserving Off-Policy Evaluation
Many reinforcement learning applications involve the use of data that is sensitive, such as medical records of patients or financial information.
A Block Coordinate Ascent Algorithm for Mean-Variance Optimization
Risk management in dynamic decision problems is a primary concern in many fields, including financial investment, autonomous driving, and healthcare.
