Search Results for author:
Found 43 papers, 26 papers with code
Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
To train an ITDA, we greedily construct a dictionary of language model activations on a dataset of prompts, selecting those activations which were worst approximated by matching pursuit on the existing dictionary.
Towards eliciting latent knowledge from LLMs with mechanistic interpretability
This work aims to be a step towards addressing the crucial problem of eliciting secret knowledge from language models, thereby contributing to their safe and reliable deployment.
Scaling sparse feature circuit finding for in-context learning
In this work, we demonstrate their effectiveness by using SAEs to deepen our understanding of the mechanism behind in-context learning (ICL).
Robustly identifying concepts introduced during chat fine-tuning using crosscoders
Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models, allowing us to track how concepts shift or emerge during fine-tuning.
An Approach to Technical AGI Safety and Security
Second, system-level security measures such as monitoring and access control can mitigate harm even if the model is misaligned.
Learning Multi-Level Features with Matryoshka Sparse Autoencoders
This organizes features hierarchically - the smaller dictionaries learn general concepts, while the larger dictionaries learn more specific concepts, without incentive to absorb the high-level features.
SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
We introduce SAEBench, a comprehensive evaluation suite that measures SAE performance across seven diverse metrics, spanning interpretability, feature disentanglement and practical applications like unlearning.
Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
Chain-of-Thought (CoT) reasoning has significantly advanced state-of-the-art AI capabilities.
Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations.
Sparse Autoencoders Do Not Find Canonical Units of Analysis
Using meta-SAEs -- SAEs trained on the decoder matrix of another SAE -- we find that latents in SAEs often decompose into combinations of latents from a smaller SAE, showing that larger SAE latents are not atomic.
Open Problems in Mechanistic Interpretability
Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals.
BatchTopK Sparse Autoencoders
A popular approach is the TopK SAE, that uses a fixed number of the most active latents per sample to reconstruct the model activations.
Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks
Sparse Autoencoders (SAEs) are an interpretability technique aimed at decomposing neural network activations into interpretable units.
Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem.
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
We primarily train SAEs on the Gemma 2 pre-trained models, but additionally release SAEs trained on instruction-tuned Gemma 2 9B for comparison.
Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
To be useful for downstream tasks, SAEs need to decompose LM activations faithfully; yet to be interpretable the decomposition must be sparse -- two objectives that are in tension.
Interpreting Attention Layer Outputs with Sparse Autoencoders
Sparse autoencoders (SAEs) are a popular method for decomposing the internal activations of trained transformers into sparse, interpretable features, and have been applied to MLP layers and the residual stream.
Confidence Regulation Neurons in Language Models
Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored.
Transcoders Find Interpretable LLM Feature Circuits
We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers.
Refusal in Language Models Is Mediated by a Single Direction
In this work, we show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size.
Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
Finally, we observe two qualitative phenomena in SAE training: feature occlusion (where a causally relevant concept is robustly overshadowed by even slightly higher-magnitude ones in the learned features), and feature over-splitting (where binary features split into many smaller, less interpretable features).
Improving Dictionary Learning with Gated Sparse Autoencoders
Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of LM activations.
How to use and interpret activation patching
Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results.
AtP*: An efficient and scalable method for localizing LLM behaviour to components
We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives.
Explorations of Self-Repair in Language Models
Prior interpretability research studying narrow distributions has preliminarily identified self-repair, a phenomena where if components in large language models are ablated, later components will change their behavior to compensate.
Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs
How do transformer-based large language models (LLMs) store and retrieve knowledge?
Universal Neurons in GPT2 Language Models
In other words, are neural mechanisms universal across different models?
Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching
We demonstrate this phenomenon in a distilled mathematical example, in two real-world domains (the indirect object identification task and factual recall), and present evidence for its prevalence in practice.
Training Dynamics of Contextual N-Grams in Language Models
We show that this neuron exists within a broader contextual n-gram circuit: we find late layer neurons which recognize and continue n-grams common in German text, but which only activate if the German neuron is active.
Linear Representations of Sentiment in Large Language Models
Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs).
Copy Suppression: Comprehensively Understanding an Attention Head
We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task.
Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where localization -- identifying the important model components -- is a key step.
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
How do sequence models represent their decision-making process?
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models.
Neuron to Graph: Interpreting Language Model Neurons at Scale
Conventional methods require examination of examples with strong neuron activation and manual identification of patterns to decipher the concepts a neuron responds to.
Finding Neurons in a Haystack: Case Studies with Sparse Probing
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood.
N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models
Understanding the function of individual neurons within language models is essential for mechanistic interpretability research.
A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations
Universality is a key hypothesis in mechanistic interpretability -- that different models learn similar features and circuits when trained on similar tasks.
Progress measures for grokking via mechanistic interpretability
Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup.
In-context Learning and Induction Heads
In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i. e. decreasing loss at increasing token indices).
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants.
An Empirical Investigation of Learning from Biased Toxicity Labels
Collecting annotations from human raters often results in a trade-off between the quantity of labels one wishes to gather and the quality of these labels.
Fully General Online Imitation Learning
If we run an imitator, we probably want events to unfold similarly to the way they would have if the demonstrator had been acting the whole time.
