On the Landscape of One-hidden-layer Sparse Networks and Beyond

16 Sep 2020  ·  Dachao Lin, Ruoyu Sun, Zhihua Zhang ·

Sparse neural networks have received increasing interests due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused on dense neural networks, and our understanding of sparse networks is very limited... In this paper, we study the loss landscape of one-hidden-layer sparse networks. We first consider sparse networks with linear activations. We show that sparse linear networks can have spurious strict minima, which is in sharp contrast to dense linear networks which do not even have spurious minima. Second, we show that spurious valleys can exist for wide sparse non-linear networks. This is different from wide dense networks which do not have spurious valleys under mild assumptions. read more

PDF Abstract
No code implementations yet. Submit your code now

Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods


No methods listed for this paper. Add relevant methods here