Search Results for author:
Found 26 papers, 19 papers with code
Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small
A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis.
Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations
In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction.
Updating CLIP to Prefer Descriptions Over Captions
This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction.
ReFT: Representation Finetuning for Language Models
We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency.
pyvene: A Library for Understanding and Improving PyTorch Models via Interventions
Interventions on model-internal states are fundamental operations in many areas of AI, including model editing, steering, robustness, and interpretability.
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations
Individual neurons participate in the representation of multiple high-level concepts.
A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments
We respond to the recent paper by Makelov et al. (2023), which reviews subspace interchange intervention methods like distributed alignment search (DAS; Geiger et al. 2023) and claims that these methods potentially cause "interpretability illusions".
Linear Representations of Sentiment in Large Language Models
Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs).
Rigorously Assessing Natural Language Explanations of Neurons
Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging.
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
For in-context learning, we test InstructGPT models and find that most prompt strategies are not successful, including those using step-by-step reasoning.
Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables.
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations.
Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability
Causal abstraction provides a theoretical foundation for mechanistic interpretability, the field concerned with providing intelligible algorithms that are faithful simplifications of the known, but opaque low-level details of black box AI models.
Causal Abstraction with Soft Interventions
Causal abstraction provides a theory describing how several causal models can represent the same system at different levels of detail.
Causal Proxy Models for Concept-Based Model Explanations
The core of our proposal is the Causal Proxy Model (CPM).
CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
We introduce CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP).
Causal Distillation for Language Models
Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
Inducing Causal Structure for Interpretable Neural Networks
In IIT, we (1) align variables in a causal model (e. g., a deterministic program or Bayesian network) with representations in a neural model and (2) train the neural model to match the counterfactual behavior of the causal model on a base input when aligned representations in both models are set to be the value they would be for a source input.
Causal Abstractions of Neural Networks
Structural analysis methods (e. g., probing and feature attribution) are increasingly important tools for neural network analysis.
Dynabench: Rethinking Benchmarking in NLP
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking.
DynaSent: A Dynamic Benchmark for Sentiment Analysis
We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis.
Relational reasoning and generalization using non-symbolic neural networks
In the two latter cases, our models perform tasks proposed in previous work to demarcate human-unique symbolic abilities.
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
We address whether neural models for Natural Language Inference (NLI) can learn the compositional interactions between lexical entailment and negation, using four methods: the behavioral evaluation methods of (1) challenge test sets and (2) systematic generalization tasks, and the structural evaluation methods of (3) probes and (4) interventions.
Posing Fair Generalization Tasks for Natural Language Inference
Deep learning models for semantics are generally evaluated using naturalistic corpora.
Recursive Routing Networks: Learning to Compose Modules for Language Understanding
The model jointly optimizes the parameters of the functions and the meta-learner{'}s policy for routing inputs through those functions.
Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity.
