Network Degeneracy as an Indicator of Training Performance: Comparing Finite and Infinite Width Angle Predictions

no code implementations • 2 Jun 2023 • Cameron Jakub, Mihai Nica

We observe this degeneracy in the sense that on initialization, inputs tend to become more and more correlated as they travel through the layers of the network.

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization

no code implementations • 20 Feb 2023 • Cameron Jakub, Mihai Nica

In this paper, we examine the evolution of the angle between two inputs to a ReLU neural network as a function of the number of layers.