no code implementations • 3 Oct 2024 • George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, Daniel Murfet
We introduce refined variants of the Local Learning Coefficient (LLC), a measure of model complexity grounded in singular learning theory, to study the development of internal structure in transformer language models during training.
1 code implementation • 6 Feb 2024 • Zach Furman, Edmund Lau
The \textit{local learning coefficient} (LLC) is a principled way of quantifying model complexity, originally derived in the context of Bayesian statistics using singular learning theory (SLT).
1 code implementation • 23 Aug 2023 • Edmund Lau, Zach Furman, George Wang, Daniel Murfet, Susan Wei
The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs).
2 code implementations • 14 Mar 2023 • Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt
We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer.