Search Results for author:
Found 43 papers, 28 papers with code
Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
For example, the LLM red-teaming literature has produced a wide variety of 'jailbreaking' techniques to elicit harmful text from models that were fine-tuned to be harmless.
When Do Universal Image Jailbreaks Transfer Between Vision-Language Models?
These results stand in stark contrast to existing evidence of universal and transferable text jailbreaks against language models and transferable adversarial attacks against image classifiers, suggesting that VLMs may be more robust to gradient-based transfer attacks.
Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments.
Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%.
Debating with More Persuasive LLMs Leads to More Truthful Answers
In anticipation of this, we ask: can weaker models assess the correctness of stronger models?
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it).
Towards Evaluating AI Systems for Moral Status Using Self-Reports
We hope our discussion inspires philosophers and AI researchers to criticize and improve our proposed methodology, as well as to run experiments to test whether self-reports can be made reliable enough to provide information about states of moral significance.
Specific versus General Principles for Constitutional AI
Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles.
Towards Understanding Sycophancy in Language Models
Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
We find that VLM-RMs are remarkably robust as long as the VLM is large enough.
Studying Large Language Model Generalization with Influence Functions
When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior?
Measuring Faithfulness in Chain-of-Thought Reasoning
Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i. e., its process for answering the question).
Question Decomposition Improves the Faithfulness of Model-Generated Reasoning
As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior.
Inverse Scaling: When Bigger Isn't Better
Here, we present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale, e. g., due to flaws in the training objective and data.
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e. g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations.
Training Language Models with Language Feedback at Scale
Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input.
Improving Code Generation by Training with Natural Language Feedback
The potential for pre-trained large language models (LLMs) to use natural language feedback at inference time has been an exciting recent development.
Pretraining Language Models with Human Preferences
Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more.
The Capacity for Moral Self-Correction in Large Language Models
We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination.
Discovering Language Model Behaviors with Model-Written Evaluations
We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse.
Constitutional AI: Harmlessness from AI Feedback
In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences.
Measuring Progress on Scalable Oversight for Large Language Models
Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand.
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs.
Few-shot Adaptation Works with UnpredicTable Data
Finetuning on the resulting dataset leads to improved FSL performance on Natural Language Processing (NLP) tasks, but not proportionally to dataset scale.
Language Models (Mostly) Know What They Know
We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly.
RL with KL penalties is better viewed as Bayesian inference
We show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function.
Training Language Models with Language Feedback
We learn from language feedback on model outputs using a three-step learning algorithm.
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
We use long contexts -- humans familiar with the context write convincing explanations for pre-selected correct and incorrect answers, and we test if those explanations allow humans who have not read the full context to more accurately determine the correct answer.
True Few-Shot Learning with Language Models
Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning.
Case-based Reasoning for Natural Language Queries over Knowledge Bases
It is often challenging to solve a complex problem from scratch, but much easier if we can access other similar problems with their solutions -- a paradigm known as case-based reasoning (CBR).
Knowledge Base Question Answering
Natural Language Queries
+1
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
We introduce a method to determine if a certain capability helps to achieve an accurate model of given data.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
Ranked #4 on
Question Answering
on WebQuestions
Unsupervised Question Decomposition for Question Answering
We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering.
Finding Generalizable Evidence by Learning to Convince Q\&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.
Finding Generalizable Evidence by Learning to Convince Q&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.
Supervised Multimodal Bitransformers for Classifying Images and Text
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks.
Ranked #1 on
Natural Language Inference
on V-SNLI
(using extra training data)
ELI5: Long Form Question Answering
We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions.
Visual Reasoning with Multi-hop Feature Modulation
Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue.
HoME: a Household Multimodal Environment
We introduce HoME: a Household Multimodal Environment for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context.
FiLM: Visual Reasoning with a General Conditioning Layer
We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation.
Ranked #3 on
Visual Question Answering (VQA)
on CLEVR-Humans
Image Retrieval with Multi-Modal Query
Visual Question Answering (VQA)
+1
Learning Visual Reasoning Without Strong Priors
Previous work has operated under the assumption that visual reasoning calls for a specialized architecture, but we show that a general architecture with proper conditioning can learn to visually reason effectively.
Semi-Supervised Learning with the Deep Rendering Mixture Model
Semi-supervised learning algorithms reduce the high cost of acquiring labeled training data by using both labeled and unlabeled data during learning.
