Search Results for author:
Found 55 papers, 23 papers with code
Inferring Reward Functions from Demonstrators with Unknown Biases
Our goal is to infer reward functions from demonstrations.
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
We primarily train SAEs on the Gemma 2 pre-trained models, but additionally release SAEs trained on instruction-tuned Gemma 2 9B for comparison.
Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making
In this paper, we build on prior work in contrastive learning and quasimetrics to show how successor features learned by contrastive learning (after a change of variables) form a temporal distance that does satisfy the triangle inequality, even in stochastic settings.
Adversaries Can Misuse Combinations of Safe Models
In this work, we show that individually testing models for misuse is inadequate; adversaries can misuse combinations of models even when each individual model is safe.
Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation
Adaptive brain stimulation can treat neurological conditions such as Parkinson's disease and post-stroke motor deficits by influencing abnormal neural activity.
AI Alignment with Changing and Influenceable Reward Functions
Existing AI alignment approaches assume that preferences are static, which is unrealistic: our preferences change, and may even be influenced by our interactions with AI systems themselves.
CoS: Enhancing Personalization and Mitigating Bias with Context Steering
Proper usage of the context enables the LLM to generate personalized responses, whereas inappropriate contextual influence can lead to stereotypical and potentially harmful generations (e. g. associating "female" with "housekeeper").
Evaluating Frontier Models for Dangerous Capabilities
To understand the risks posed by a new AI system, we must understand what it can and cannot do.
A Generalized Acquisition Function for Preference-based Reward Learning
Preference-based reward learning is a popular technique for teaching robots and autonomous systems how a human user wants them to perform a task.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we introduce the Gemini 1. 5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio.
Ranked #1 on
Zero-Shot Video Question Answer
on Video-MME
Preventing Reward Hacking with Occupancy Measure Regularization
Thus, we propose regularizing based on the OM divergence between policies instead of AD divergence to prevent reward hacking.
When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment.
The Effective Horizon Explains Deep RL Performance in Stochastic Environments
Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks.
Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations
LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they are not trained to optimize for overall conversational outcomes after multiple turns of interaction.
Offline RL with Observation Histories: Analyzing and Improving Sample Complexity
Theoretically, we show that standard offline RL algorithms conditioned on observation histories suffer from poor sample complexity, in accordance with the above intuition.
Managing extreme AI risks amid rapid progress
Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals.
Learning Optimal Advantage from Preferences and Mistaking it for Reward
Most recent work assumes that human preferences are generated based only upon the reward accrued within those segments, or their partial return.
Learning to Model the World with Language
While current agents can learn to execute simple language instructions, we aim to build agents that leverage diverse language -- language like "this button turns on the TV" or "I put the bowls away" -- that conveys general knowledge, describes the state of the world, provides interactive feedback, and more.
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals.
Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control
Our method achieves robust performance in the real world by learning an embedding from the labeled data that aligns language not to the goal image, but rather to the desired change between the start and goal images that the instruction corresponds to.
Toward Grounded Commonsense Reasoning
We additionally illustrate our approach with a robot on 2 carefully designed surfaces.
Bridging RL Theory and Practice with the Effective Horizon
Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does.
Automatically Auditing Large Language Models via Discrete Optimization
Auditing large language models for unexpected behaviors is critical to preempt catastrophic deployments, yet remains challenging.
Towards Modeling and Influencing the Dynamics of Human Learning
Interestingly, robot actions influence what this experience is, and therefore influence how people's internal models change.
On the Sensitivity of Reward Inference to Misspecified Human Models
This begs the question: how accurate do these models need to be in order for the reward inference to be accurate?
Time-Efficient Reward Learning via Visually Assisted Cluster Ranking
One of the most successful paradigms for reward learning uses human feedback in the form of comparisons.
UniMASK: Unified Inference in Sequential Decision Problems
Randomly masking and predicting word tokens has been a successful approach in pre-training language models for a variety of downstream tasks.
Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration
We show that using optimal behavior as a prior for human models makes these models vastly more data-efficient and able to generalize to new environments.
Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers
Randomly masking and predicting word tokens has been a successful approach in pre-training language models for a variety of downstream tasks.
Estimating and Penalizing Induced Preference Shifts in Recommender Systems
These steps involve two challenging ingredients: estimation requires anticipating how hypothetical algorithms would influence user preferences if deployed - we do this by using historical user interaction data to train a predictive user model which implicitly contains their preference dynamics; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted - we use the notion of "safe shifts", that define a trust region within which behavior is safe: for instance, the natural way in which users would shift without interference from the system could be deemed "safe".
The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models
However, these models fail when humans exhibit systematic suboptimality, i. e. when their deviations from optimal behavior are not independent, but instead consistent over time.
Inferring Rewards from Language in Context
In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e. g., selecting that flight).
Human irrationality: both bad and good for reward inference
More importantly, we show that an irrational human, when correctly modelled, can communicate more information about the reward than a perfectly rational human can.
B-Pref: Benchmarking Preference-Based Reinforcement Learning
However, it is difficult to quantify the progress in preference-based RL due to the lack of a commonly adopted benchmark.
The MineRL BASALT Competition on Learning from Human Feedback
Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve.
Learning What To Do by Simulating the Past
Since reward functions are hard to specify, recent work has focused on learning policies from human feedback.
Choice Set Misspecification in Reward Inference
A promising alternative to manually specifying reward functions is to enable robots to infer them from human feedback, like demonstrations or corrections.
The impacts of known and unknown demonstrator irrationality on reward inference
Surprisingly, we find that if we give the learner access to the correct model of the demonstrator's irrationality, these irrationalities can actually help reward inference.
Benefits of Assistance over Reward Learning
By merging reward learning and control, assistive agents can reason about the impact of control actions on reward learning, leading to several advantages over agents based on reward learning.
XT2: Training an X-to-Text Typing Interface with Online Learning from Implicit Feedback
In the typing domain, we leverage backspaces as implicit feedback that the interface did not perform the desired action.
AvE: Assistance via Empowerment
One difficulty in using artificial agents for human-assistive applications lies in the challenge of accurately assisting with a person's goal(s).
On the Utility of Learning about Humans for Human-AI Coordination
While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves.
Few-Shot Intent Inference via Meta-Inverse Reinforcement Learning
A significant challenge for the practical application of reinforcement learning toreal world problems is the need to specify an oracle reward function that correctly defines a task.
Preferences Implicit in the State of the World
We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized.
The Assistive Multi-Armed Bandit
Learning preferences implicit in the choices humans make is a well studied problem in both economics and computer science.
On the Utility of Model Learning in HRI
Fundamental to robotics is the debate between model-based and model-free learning: should the robot build an explicit model of the world, or learn a policy directly?
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
A significant challenge for the practical application of reinforcement learning in the real world is the need to specify an oracle reward function that correctly defines a task.
Inverse Reward Design
When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios.
Should Robots be Obedient?
We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order.
Translating Neuralese
Several approaches have recently been proposed for learning decentralized deep multiagent policies that coordinate via a differentiable communication channel.
DART: Noise Injection for Robust Imitation Learning
One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy.
The Off-Switch Game
We analyze a simple game between a human H and a robot R, where H can press R's off switch but R can disable the off switch.
Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrations
Although policies learned with RC sampling can be superior to HC sampling for standard learning models such as linear SVMs, policies learned with HC sampling may be comparable with highly-expressive learning models such as deep learning and hyper-parametric decision trees, which have little model error.
Cooperative Inverse Reinforcement Learning
For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans.
