Depth Separation in Norm-Bounded Infinite-Width Neural Networks

no code implementations • 13 Feb 2024 • Suzanna Parkinson, Greg Ongie, Rebecca Willett, Ohad Shamir, Nathan Srebro

We also show that a similar statement in the reverse direction is not possible: any function learnable with polynomial sample complexity by a norm-controlled depth-2 ReLU network with infinite width is also learnable with polynomial sample complexity by a norm-controlled depth-3 ReLU network.

Linear Neural Network Layers Promote Learning Single- and Multiple-Index Models

no code implementations • 24 May 2023 • Suzanna Parkinson, Greg Ongie, Rebecca Willett

This paper explores the implicit bias of overparameterized neural networks of depth greater than two layers.