Search Results for author:
Found 25 papers, 8 papers with code
Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling
This raises a critical question: Do the established principles that proved successful in the generalization-centric era remain valid in this new era of scaling?
Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
We find that for a fixed dataset, larger and longer-trained LMs hallucinate less.
Scaling Exponents Across Parameterizations and Optimizers
Robust and effective scaling of models from small to large width typically requires the precise adjustment of many algorithmic and architectural details, such as parameterization and optimizer choices.
4+3 Phases of Compute-Optimal Neural Scaling Laws
We consider the solvable neural scaling model with three parameters: data complexity, target complexity, and model-parameter-count.
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times.
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment.
Small-scale proxies for large-scale Transformer training instabilities
In this work, we seek ways to reproduce and study training stability and instability at smaller scales.
Fast Neural Kernel Embeddings for General Activations
Moreover, most prior works on neural kernels have focused on the ReLU activation, mainly due to its popularity but also due to the difficulty of computing such kernels for general activations.
Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm
Although learning in high dimensions is commonly believed to suffer from the curse of dimensionality, modern machine learning methods often exhibit an astonishing power to tackle a wide range of challenging real-world learning problems without using abundant amounts of data.
Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improvements under different model and data scaling regimes.
Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks
It is well-known that the eigenstructure of infinite-width multilayer perceptrons (MLPs) depends solely on the concept frequency, which measures the order of interactions.
Dataset Distillation with Infinitely Wide Convolutional Networks
The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data.
What Breaks The Curse of Dimensionality in Deep Learning?
By computing an eigen-decomposition of the infinite-width limits (aka Neural Kernels) of these architectures, we characterize how inductive biases (locality, weight-sharing, pooling, etc) and the breaking of spurious symmetries can affect the performance of these learning systems.
Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Finite Versus Infinite Neural Networks: an Empirical Study
We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods.
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
Modern neural networks are often regarded as complex black-box functions whose behavior is difficult to understand owing to their nonlinear dependence on the data and the nonconvexity in their loss landscapes.
Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance.
Disentangling Trainability and Generalization in Deep Neural Networks
A longstanding goal in the theory of deep learning is to characterize the conditions under which a given neural network architecture will be trainable, and if so, how well it might generalize to unseen data.
Neural Tangents: Fast and Easy Infinite Neural Networks in Python
Neural Tangents is a library designed to enable research into infinite-width neural networks.
Disentangling Trainability and Generalization in Deep Learning
In this paper, we discuss these challenging issues in the context of wide neural networks at large depths where we will see that the situation simplifies considerably.
Deep Bayesian Convolutional Networks with Many Channels are Gaussian Processes
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
A longstanding goal in deep learning research has been to precisely characterize training and generalization.
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
In this work, we demonstrate that it is possible to train vanilla CNNs with ten thousand layers or more simply by using an appropriate initialization scheme.
