Search Results for author:
Found 28 papers, 14 papers with code
Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
Sparse Autoencoders (SAEs) are widely used to interpret neural networks by identifying meaningful concepts from their representations.
Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models
Sparse Autoencoders (SAEs) have emerged as a powerful framework for machine learning interpretability, enabling the unsupervised decomposition of model representations into a dictionary of abstract, human-interpretable concepts.
ICLR: In-Context Learning of Representations
Specifically, if we provide in-context exemplars wherein a concept plays a different role than what the pretraining data suggests, do models reorganize their representations in accordance with these novel semantics?
Competition Dynamics Shape Algorithmic Phases of In-Context Learning
In-Context Learning (ICL) has significantly expanded the general-purpose nature of large language models, allowing them to adapt to novel tasks using merely the inputted context.
Abrupt Learning in Transformers: A Case Study on Matrix Completion
We also analyze the training dynamics of individual model components to understand the sudden drop in loss.
Towards Reliable Evaluation of Behavior Steering Interventions in LLMs
Representation engineering methods have recently shown promise for enabling efficient steering of model behavior.
Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing
We call this phenomenon representation shattering and demonstrate that it results in degradation of factual recall and reasoning performance more broadly.
Analyzing (In)Abilities of SAEs via Formal Languages
Autoencoders have been used for finding interpretable and disentangled features underlying neural network representations in both image and text domains.
Dynamics of Concept Learning and Compositional Generalization
We mathematically analyze the learning dynamics of neural networks trained on this SIM task and show that, despite its simplicity, SIM's learning dynamics capture and help explain key empirical observations on compositional generalization with diffusion models identified in prior work.
A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language
We empirically investigate this definition by proposing an experimental system grounded in a context-sensitive formal language and find that Transformers trained to perform tasks on top of strings from this language indeed exhibit emergent capabilities.
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment.
Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space
Modern generative models demonstrate impressive capabilities, likely stemming from an ability to identify and manipulate abstract concepts underlying their training data.
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs).
Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model
Stepwise inference protocols, such as scratchpads and chain-of-thought, help language models solve complex problems by decomposing them into a sequence of simpler subproblems.
FoMo Rewards: Can we cast foundation models as reward functions?
We explore the viability of casting foundation models as generic reward functions for reinforcement learning.
Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks
Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy.
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks
Transformers trained on huge text corpora exhibit a remarkable set of capabilities, e. g., performing basic arithmetic.
In-Context Learning Dynamics with Random Binary Sequences
Large language models (LLMs) trained on huge corpora of text datasets demonstrate intriguing capabilities, achieving state-of-the-art performance on tasks they were not explicitly trained for.
Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
Motivated by this, we perform a controlled study for understanding compositional generalization in conditional diffusion models in a synthetic setting, varying different attributes of the training data and measuring the model's ability to generate samples out-of-distribution.
Mechanistic Mode Connectivity
We study neural network loss landscapes through the lens of mode connectivity, the observation that minimizers of neural networks retrieved via training on a dataset are connected via simple paths of low loss.
What shapes the loss landscape of self-supervised learning?
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL).
Analyzing Data-Centric Properties for Graph Contrastive Learning
Overall, our work rigorously contextualizes, both empirically and theoretically, the effects of data-centric properties on augmentation strategies and learning paradigms for graph SSL.
Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering
Federated learning is generally used in tasks where labels are readily available (e. g., next word prediction).
Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices
Unsupervised graph representation learning is critical to a wide range of applications where labels may be scarce or expensive to procure.
Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning
Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning.
How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation
Catastrophic forgetting undermines the effectiveness of deep neural networks (DNNs) in scenarios such as continual learning and lifelong learning.
A Gradient Flow Framework For Analyzing Network Pruning
We use this framework to determine the relationship between pruning measures and evolution of model parameters, establishing several results related to pruning models early-on in training: (i) magnitude-based pruning removes parameters that contribute least to reduction in loss, resulting in models that converge faster than magnitude-agnostic methods; (ii) loss-preservation based pruning preserves first-order model evolution dynamics and is therefore appropriate for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gradient norm via pruning can produce poorly performing models.
OrthoReg: Robust Network Pruning Using Orthonormality Regularization
To address this issue, we propose OrthoReg, a principled regularization strategy that enforces orthonormality on a network's filters to reduce inter-filter correlation, thereby allowing reliable, efficient determination of group importance estimates, improved trainability of pruned networks, and efficient, simultaneous pruning of large groups of filters.
