Search Results for author:
Found 24 papers, 8 papers with code
Inferring Reward Functions from Demonstrators with Unknown Biases
Our goal is to infer reward functions from demonstrations.
Evaluating Frontier Models for Dangerous Capabilities
To understand the risks posed by a new AI system, we must understand what it can and cannot do.
AtP*: An efficient and scalable method for localizing LLM behaviour to components
We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives.
Challenges with unsupervised LLM knowledge discovery
We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent.
BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks
Given the completion of two years of BASALT competitions, we offer to the community a formalized benchmark through the BASALT Evaluation and Demonstrations Dataset (BEDD), which serves as a resource for algorithm development and performance assessment.
Explaining grokking through circuit efficiency
One of the most surprising puzzles in neural network generalisation is grokking: a network with perfect training accuracy but poor generalisation will, upon further training, transition to perfect generalisation.
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models.
Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition
To facilitate research in the direction of fine-tuning foundation models from human feedback, we held the MineRL BASALT Competition on Fine-Tuning from Human Feedback at NeurIPS 2022.
SIRL: Similarity-based Implicit Representation Learning
This, in turn, is what enables the robot to disambiguate between what needs to go into the representation versus what is spurious, as well as what aspects of behavior can be compressed together versus not.
Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
However, an AI system may pursue an undesired goal even when the specification is correct, in the case of goal misgeneralization.
An Empirical Investigation of Representation Learning for Imitation
We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation across several environment suites.
Retrospective on the 2021 BASALT Competition on Learning from Human Feedback
The goal of the competition was to promote research towards agents that use learning from human feedback (LfHF) techniques to solve open-world tasks.
The MineRL BASALT Competition on Learning from Human Feedback
Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve.
Learning What To Do by Simulating the Past
Since reward functions are hard to specify, recent work has focused on learning policies from human feedback.
Combining Reward Information from Multiple Sources
We study this problem in the setting with two conflicting reward functions learned from different sources.
Choice Set Misspecification in Reward Inference
A promising alternative to manually specifying reward functions is to enable robots to infer them from human feedback, like demonstrations or corrections.
Evaluating the Robustness of Collaborative Agents
We apply this methodology to build a suite of unit tests for the Overcooked-AI environment, and use this test suite to evaluate three proposals for improving robustness.
Benefits of Assistance over Reward Learning
By merging reward learning and control, assistive agents can reason about the impact of control actions on reward learning, leading to several advantages over agents based on reward learning.
The MAGICAL Benchmark for Robust Imitation
This rewards precise reproduction of demonstrations in one particular environment, but provides little information about how robustly an algorithm can generalise the demonstrator's intent to substantially different deployment settings.
Optimal Policies Tend to Seek Power
Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives.
On the Utility of Learning about Humans for Human-AI Coordination
While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves.
On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
But in the era of deep learning, a natural suggestion researchers make is to avoid mathematical models of human behavior that are fraught with specific assumptions, and instead use a purely data-driven approach.
Preferences Implicit in the State of the World
We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized.
Active Inverse Reward Design
We propose structuring this process as a series of queries asking the user to compare between different reward functions.
