Search Results for author:
Found 43 papers, 17 papers with code
The SaTML '24 CNN Interpretability Competition: New Innovations for Concept-Level Interpretability
Interpretability techniques are valuable for helping humans understand and oversee AI systems.
Defending Against Unforeseen Failure Modes with Latent Adversarial Training
Despite extensive diagnostics and debugging by developers, AI systems sometimes exhibit harmful unintended behaviors.
Eight Methods to Evaluate Robust Unlearning in LLMs
Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it.
Black-Box Access is Insufficient for Rigorous AI Audits
The effectiveness of an audit, however, depends on the degree of system access granted to auditors.
Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
We prove that standard applications of preference learning, including reinforcement learning from human feedback (RLHF), implicitly aggregate over hidden contexts according to a well-known voting rule called Borda count.
Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?
This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents.
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals.
Measuring the Success of Diffusion Models at Imitating Human Artists
When Stable Diffusion is prompted to imitate an artist from this set, we find that the artist can be identified from the imitation with an average accuracy of 81. 0%.
Explore, Establish, Exploit: Red Teaming Language Models from Scratch
Using a pre-existing classifier does not allow for red-teaming to be tailored to the target model.
Recommending to Strategic Users
We model this user behavior as a two-stage noisy signalling game between the recommendation system and users: the recommendation system initially commits to a recommendation policy, presents content to the users during a cold start phase which the users choose to strategically consume in order to affect the types of content they will be recommended in a recommendation phase.
Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks
Some previous works have proposed using human-interpretable adversarial attacks including copy/paste attacks in which one natural image pasted into another causes an unexpected misclassification.
Red Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents
In this work, we study white-box adversarial policies and show that having access to a target agent's internal state can be useful for identifying its vulnerabilities.
Formal Contracts Mitigate Social Dilemmas in Multi-Agent RL
In this work, we draw upon the idea of formal contracting from economics to overcome diverging incentives between agents in MARL.
Towards Psychologically-Grounded Dynamic Preference Models
Second, we discuss implications of dynamic preference models for recommendation systems evaluation and design.
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
The last decade of machine learning has seen drastic increases in scale and capabilities.
Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
We collect a set of values that seem most relevant to recommender systems operating across different domains, then examine them from the perspectives of current industry practice, measurement, product design, and policy approaches.
How to talk so AI will learn: Instructions, descriptions, and autonomy
We study two distinct types of language: $\textit{instructions}$, which provide information about the desired policy, and $\textit{descriptions}$, which provide information about the reward function.
Estimating and Penalizing Induced Preference Shifts in Recommender Systems
These steps involve two challenging ingredients: estimation requires anticipating how hypothetical algorithms would influence user preferences if deployed - we do this by using historical user interaction data to train a predictive user model which implicitly contains their preference dynamics; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted - we use the notion of "safe shifts", that define a trust region within which behavior is safe: for instance, the natural way in which users would shift without interference from the system could be deemed "safe".
Linguistic communication as (inverse) reward design
We then define a pragmatic listener which performs inverse reward design by jointly inferring the speaker's latent horizon and rewards.
Guided Imitation of Task and Motion Planning
While modern policy optimization methods can do complex manipulation from sensory data, they struggle on problems with extended time horizons and multiple sub-goals.
Robust Feature-Level Adversaries are Interpretability Tools
We demonstrate that they can be used to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale.
What are you optimizing for? Aligning Recommender Systems with Human Values
We describe cases where real recommender systems were modified in the service of various human values such as diversity, fairness, well-being, time well spent, and factual accuracy.
Consequences of Misaligned AI
We consider the cost of this incompleteness by analyzing a model of a principal and an agent in a resource constrained world where the $L$ attributes of the state correspond to different sources of utility for the principal.
Multi-Principal Assistance Games: Definition and Collegial Mechanisms
We introduce the concept of a multi-principal assistance game (MPAG), and circumvent an obstacle in social choice theory, Gibbard's theorem, by using a sufficiently collegial preference inference mechanism.
Multi-Principal Assistance Games
Assistance games (also known as cooperative inverse reinforcement learning games) have been proposed as a model for beneficial AI, wherein a robotic agent must act on behalf of a human principal but is initially uncertain about the humans payoff function.
Silly rules improve the capacity of agents to learn stable enforcement and compliance behaviors
How can societies learn to enforce and comply with social norms?
An Extensible Interactive Interface for Agent Design
In deep reinforcement learning, for example, directly specifying a reward as a function of a high-dimensional observation is challenging.
Adversarial Training with Voronoi Constraints
We show that adversarial training with Voronoi constraints produces robust models which significantly improve over the state-of-the-art on MNIST and are competitive on CIFAR-10.
Conservative Agency via Attainable Utility Preservation
Reward functions are easy to misspecify; although designers can make corrections after observing mistakes, an agent pursuing a misspecified reward function can irreversibly change the state of its environment.
The Assistive Multi-Armed Bandit
Learning preferences implicit in the choices humans make is a well studied problem in both economics and computer science.
On the Utility of Model Learning in HRI
Fundamental to robotics is the debate between model-based and model-free learning: should the robot build an explicit model of the world, or learn a policy directly?
Human-AI Learning Performance in Multi-Armed Bandits
People frequently face challenging decision-making problems in which outcomes are uncertain or unknown.
Legible Normativity for AI Alignment: The Value of Silly Rules
It has become commonplace to assert that autonomous agents will have to be built to follow human rules of behavior--social norms and laws.
On the Geometry of Adversarial Examples
Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models.
Active Inverse Reward Design
We propose structuring this process as a series of queries asking the user to compare between different reward functions.
An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning
We apply this update to a variety of POMDP solvers and find that it enables us to scale CIRL to non-trivial problems, with larger reward parameter spaces, and larger action spaces for both robot and human.
Simplifying Reward Design through Divide-and-Conquer
Designing a good reward function is essential to robot planning and reinforcement learning, but it can also be challenging and frustrating.
Incomplete Contracting and AI Alignment
We suggest that the analysis of incomplete contracting developed by law and economics researchers can provide a useful framework for understanding the AI alignment problem and help to generate a systematic approach to finding solutions.
Inverse Reward Design
When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios.
Pragmatic-Pedagogic Value Alignment
In robotics, value alignment is key to the design of collaborative robots that can integrate into human workflows, successfully inferring and adapting to their users' objectives as they go.
Should Robots be Obedient?
We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order.
The Off-Switch Game
We analyze a simple game between a human H and a robot R, where H can press R's off switch but R can disable the off switch.
Cooperative Inverse Reinforcement Learning
For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans.
