Neural Network Compression Using Higher-Order Statistics and AuxiliaryReconstruction Losses

In this paper, the problem of pruning and compressingthe weights of various layers of deep neural networks is in-vestigated. The proposed method aims to remove redundantfilters from the network to reduce computational complex-ity and storage requirements, while improving the perfor-mance of the original network. More specifically, a novelfilter selection criterion is introduced based on the fact thatfilters whose weights follow a Gaussian distribution corre-spond to hidden units that do not capture important aspectsof data. To this end, Higher Order Statistics (HOS) areused and filters with low cumulant values that do not de-viate significantly from Gaussian distribution are identifiedand removed from the network. In addition, a novel prun-ing strategy is proposed aiming to decide on the pruningratio of each layer using the Shapiro-Wilk normality test.The use of auxiliary MSE losses (intermediate and afterthe softmax layer) during the fine-tuning phase further im-proves the overall performance of the compressed network.Extensive experiments with different network architecturesand comparison with state-of-the-art approaches on well-known public datasets, such as CIFAR-10, CIFAR-100 andILSCVR-12, demonstrate the great potential of the proposedapproach.