Search Results for author:
Found 21 papers, 16 papers with code
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models.
On the Origins of the Block Structure Phenomenon in Neural Network Representations
Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.
A Survey of Deep Learning for Scientific Discovery
Over the past few years, we have seen fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks.
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.
Insights on representational similarity in neural networks with canonical correlation
Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, even across groups of networks trained on identical tasks, and over the course of training.
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods).
Do Vision Transformers See Like Convolutional Neural Networks?
Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.
Exponential expressivity in deep neural networks through transient chaos
We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights.
Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures
More generally, the paper considers the learning of logical functions with gradient descent (GD) on neural networks.
Adversarial Spheres
We hypothesize that this counter intuitive behavior is a naturally occurring result of the high dimensional geometry of the data manifold.
Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?
Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization.
The Algorithmic Automation Problem: Prediction, Triage, and Human Effort
In a wide array of areas, algorithms are matching and surpassing the performance of human experts, leading to consideration of the roles of human judgment and algorithmic prediction in these domains.
Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
In PVR, this is done by having one part of the task input act as a pointer, giving instructions on a different input location, which forms the output.
Teaching with Commentaries
We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process.
Transfusion: Understanding Transfer Learning for Medical Imaging
Investigating the learned representations and features, we find that some of the differences from transfer learning are due to the over-parametrization of standard models rather than sophisticated feature reuse.
On the Expressive Power of Deep Neural Networks
We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute.
Survey of Expressivity in Deep Neural Networks
This quantity grows exponentially in the depth of the network, and is responsible for the depth sensitivity observed.
Direct Uncertainty Prediction for Medical Second Opinions
Our central methodological finding is that Direct Uncertainty Prediction (DUP), training a model to predict an uncertainty score directly from the raw patient features, works better than Uncertainty Via Classification, the two-step process of training a classifier and postprocessing the output distribution to give an uncertainty score.
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks.
Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations
Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.
