Search Results for author:
Found 21 papers, 12 papers with code
How to use and interpret activation patching
Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results.
AtP*: An efficient and scalable method for localizing LLM behaviour to components
We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives.
Explorations of Self-Repair in Language Models
Prior interpretability research studying narrow distributions has preliminarily identified self-repair, a phenomena where if components in large language models are ablated, later components will change their behavior to compensate.
Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs
How do transformer-based large language models (LLMs) store and retrieve knowledge?
Universal Neurons in GPT2 Language Models
In other words, are neural mechanisms universal across different models?
Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching
We demonstrate this phenomenon in a distilled mathematical example, in two real-world domains (the indirect object identification task and factual recall), and present evidence for its prevalence in practice.
Training Dynamics of Contextual N-Grams in Language Models
We show that this neuron exists within a broader contextual n-gram circuit: we find late layer neurons which recognize and continue n-grams common in German text, but which only activate if the German neuron is active.
Linear Representations of Sentiment in Large Language Models
Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs).
Copy Suppression: Comprehensively Understanding an Attention Head
We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task.
Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where localization -- identifying the important model components -- is a key step.
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
How do sequence models represent their decision-making process?
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models.
Neuron to Graph: Interpreting Language Model Neurons at Scale
Conventional methods require examination of examples with strong neuron activation and manual identification of patterns to decipher the concepts a neuron responds to.
Finding Neurons in a Haystack: Case Studies with Sparse Probing
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood.
N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models
Understanding the function of individual neurons within language models is essential for mechanistic interpretability research.
A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations
Universality is a key hypothesis in mechanistic interpretability -- that different models learn similar features and circuits when trained on similar tasks.
Progress measures for grokking via mechanistic interpretability
Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup.
In-context Learning and Induction Heads
In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i. e. decreasing loss at increasing token indices).
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants.
An Empirical Investigation of Learning from Biased Toxicity Labels
Collecting annotations from human raters often results in a trade-off between the quantity of labels one wishes to gather and the quality of these labels.
Fully General Online Imitation Learning
If we run an imitator, we probably want events to unfold similarly to the way they would have if the demonstrator had been acting the whole time.
