Search Results for author:
Found 3 papers, 0 papers with code
LOFT: Finding Lottery Tickets through Filter-wise Training
\textsc{LoFT} is a model-parallel pretraining algorithm that partitions convolutional layers by filters to train them independently in a distributed setting, resulting in reduced memory and communication costs during pretraining.
How much pre-training is enough to discover a good subnetwork?
Aiming to mathematically analyze the amount of dense network pre-training needed for a pruned network to perform well, we discover a simple theoretical bound in the number of gradient descent pre-training iterations on a two-layer, fully-connected network, beyond which pruning via greedy forward selection [61] yields a subnetwork that achieves good training error.
Mitigating deep double descent by concatenating inputs
In this work, we explore the connection between the double descent phenomena and the number of samples in the deep neural network setting.
