Search Results for author:
Found 6 papers, 3 papers with code
(How) Can Transformers Predict Pseudo-Random Numbers?
Our analysis reveals that with sufficient architectural capacity and training data variety, Transformers can perform in-context prediction of LCG sequences with unseen moduli ($m$) and parameters ($a, c$).
Grokking Modular Polynomials
Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest.
Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
In this work, we study the emergence of in-context learning and skill composition in a collection of modular arithmetic tasks.
To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large.
AutoInit: Automatic Initialization via Jacobian Tuning
Good initialization is essential for training Deep Neural Networks (DNNs).
Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
We derive recurrence relations for the norms of partial Jacobians and utilize these relations to analyze criticality of deep fully connected neural networks with LayerNorm and/or residual connections.
