Search Results for author:
Found 33 papers, 16 papers with code
What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries
Deep neural network classifiers partition input space into high confidence regions for each class.
Deep equilibrium networks are sensitive to initialization statistics
Deep equilibrium networks (DEQs) are a promising way to construct models which trade off memory for compute.
Fast Finite Width Neural Tangent Kernel
We perform the first in-depth analysis of the compute and memory requirements for NTK computation in finite width networks.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Gradients are Not All You Need
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades.
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
Using an extended and formalized version of the Q/C map analysis of Poole et al. (2016), along with Neural Tangent Kernel theory, we identify the main pathologies present in deep networks that prevent them from training fast and generalizing to unseen data, and show how these can be avoided by carefully controlling the "shape" of the network's initialization-time kernel function.
Tilting the playing field: Dynamical loss functions for machine learning
In underparameterized networks, such dynamical loss functions can lead to successful training for networks that fail to find a deep minima of the standard cross-entropy loss.
Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
We show that both data whitening and second order optimization can harm or entirely prevent generalization.
Finite Versus Infinite Neural Networks: an Empirical Study
We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods.
On the infinite width limit of neural networks with a standard parameterization
However, the extrapolation of both of these parameterizations to infinite width is problematic.
Disentangling Trainability and Generalization in Deep Neural Networks
A longstanding goal in the theory of deep learning is to characterize the conditions under which a given neural network architecture will be trainable, and if so, how well it might generalize to unseen data.
JAX, M.D.: A Framework for Differentiable Physics
We introduce JAX MD, a software package for performing differentiable physics simulations with a focus on molecular dynamics.
Neural Tangents: Fast and Easy Infinite Neural Networks in Python
Neural Tangents is a library designed to enable research into infinite-width neural networks.
The Dynamics of Signal Propagation in Gated Recurrent Neural Networks
We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower.
JAX MD: End-to-End Differentiable, Hardware Accelerated, Molecular Dynamics in Pure Python
In this work we bring the substantial advances in software that have taken place in machine learning to MD with JAX, M. D.
A Mean Field Theory of Batch Normalization
We develop a mean field theory for batch normalization in fully-connected feedforward neural networks.
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
A longstanding goal in deep learning research has been to precisely characterize training and generalization.
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower.
Peptide-Spectra Matching from Weak Supervision
As in many other scientific domains, we face a fundamental problem when using machine learning to identify proteins from mass spectrometry data: large ground truth datasets mapping inputs to correct outputs are extremely difficult to obtain.
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
In this work, we demonstrate that it is possible to train vanilla CNNs with ten thousand layers or more simply by using an appropriate initialization scheme.
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
We develop a theory for signal propagation in recurrent networks after random initialization using a combination of mean field theory and random matrix theory.
Machine learning determination of atomic dynamics at grain boundaries
In polycrystalline materials, grain boundaries are sites of enhanced atomic motion, but the complexity of the atomic structures within a grain boundary network makes it difficult to link the structure and atomic dynamics.
Materials Science
The Emergence of Spectral Universality in Deep Networks
Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude.
Adversarial Spheres
We hypothesize that this counter intuitive behavior is a naturally occurring result of the high dimensional geometry of the data manifold.
Mean Field Residual Networks: On the Edge of Chaos
Classical feedforward neural networks, such as those with tanh activations, exhibit exponential behavior on the average when propagating inputs forward or gradients backward.
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed.
Intriguing Properties of Adversarial Examples
Finally, we study the effect of network architectures on adversarial sensitivity.
Deep Neural Networks as Gaussian Processes
As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network.
A Correspondence Between Random Neural Networks and Statistical Field Theory
In this work, we show that the distribution of pre-activations in random neural networks can be exactly mapped onto lattice models in statistical physics.
Combining Machine Learning and Physics to Understand Glassy Systems
Our understanding of supercooled liquids and glasses has lagged significantly behind that of simple liquids and crystalline solids.
Neural Message Passing for Quantum Chemistry
Supervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science.
Ranked #5 on
Graph Regression
on ZINC 100k
Machine learning prediction errors better than DFT accuracy
We investigate the impact of choosing regressors and molecular representations for the construction of fast machine learning (ML) models of thirteen electronic ground-state properties of organic molecules.
Ranked #18 on
Formation Energy
on QM9
Deep Information Propagation
We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks.
