Search Results for author: Zach Furman

Found 4 papers, 3 papers with code

Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient

no code implementations3 Oct 2024 George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, Daniel Murfet

We introduce refined variants of the Local Learning Coefficient (LLC), a measure of model complexity grounded in singular learning theory, to study the development of internal structure in transformer language models during training.

Learning Theory

Estimating the Local Learning Coefficient at Scale

1 code implementation6 Feb 2024 Zach Furman, Edmund Lau

The \textit{local learning coefficient} (LLC) is a principled way of quantifying model complexity, originally derived in the context of Bayesian statistics using singular learning theory (SLT).

Learning Theory

The Local Learning Coefficient: A Singularity-Aware Complexity Measure

1 code implementation23 Aug 2023 Edmund Lau, Zach Furman, George Wang, Daniel Murfet, Susan Wei

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs).

Inductive Bias Learning Theory

Eliciting Latent Predictions from Transformers with the Tuned Lens

2 code implementations14 Mar 2023 Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt

We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer.

Language Modelling

Cannot find the paper you are looking for? You can Submit a new open access paper.