Search Results for author:
Found 6 papers, 0 papers with code
Learning Associative Memories with Gradient Descent
This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings.
The Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding
Additionally, we show that, if a stationary point does not contain "escape neurons", which are defined with first-order conditions, then it must be a local minimum.
Expand-and-Cluster: Parameter Recovery of Neural Networks
Can we identify the parameters of a neural network by probing its input-output mapping?
Understanding out-of-distribution accuracies through quantifying difficulty of test samples
Existing works show that although modern neural networks achieve remarkable generalization performance on the in-distribution (ID) dataset, the accuracy drops significantly on the out-of-distribution (OOD) datasets \cite{recht2018cifar, recht2019imagenet}.
Weight-space symmetry in neural network loss landscapes revisited
In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, d$, we construct continuous paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange.
Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
The permutation symmetry of neurons in each layer of a deep neural network gives rise not only to multiple equivalent global minima of the loss function, but also to first-order saddle points located on the path between the global minima.
