Search Results for author:
Found 12 papers, 5 papers with code
PARQ: Piecewise-Affine Regularized Quantization
We develop a principled method for quantization-aware training (QAT) of large-scale machine learning models.
Spectral Journey: How Transformers Predict the Shortest Path
In this setting, the representations and the dynamics learned by the model are interpretable.
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
We then leverage tools from information theory to show that such embeddings maximize a well-known lower bound on mutual information between views, thereby connecting the geometric perspective of MMCR to the information-theoretic perspective commonly discussed in MVSSL.
Grokking Modular Polynomials
Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest.
Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
In this work, we study the emergence of in-context learning and skill composition in a collection of modular arithmetic tasks.
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs?
The Unreasonable Ineffectiveness of the Deeper Layers
We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed.
Bridging Associative Memory and Probabilistic Modeling
Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions.
To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large.
Grokking modular arithmetic
We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as ``grokking''.
AutoInit: Automatic Initialization via Jacobian Tuning
Good initialization is essential for training Deep Neural Networks (DNNs).
Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
We derive recurrence relations for the norms of partial Jacobians and utilize these relations to analyze criticality of deep fully connected neural networks with LayerNorm and/or residual connections.
