Search Results for author:
Found 7 papers, 3 papers with code
Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
We derive recurrence relations for the norms of partial Jacobians and utilize these relations to analyze criticality of deep fully connected neural networks with LayerNorm and/or residual connections.
AutoInit: Automatic Initialization via Jacobian Tuning
Good initialization is essential for training Deep Neural Networks (DNNs).
Grokking modular arithmetic
We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as ``grokking''.
To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large.
Bridging Associative Memory and Probabilistic Modeling
Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions.
The Unreasonable Ineffectiveness of the Deeper Layers
We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed.
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs?
