Spectral-Normalized Identity Priors

Spectral-Normalized Identity Priors, or SNIP, is a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. It is applicable to any structured module, including a single attention head, an entire attention block, or a feed-forward subnetwork. The method identifies and discards unimportant non-linear mappings in the residual connections by applying a thresholding operator on the function norm. Furthermore, spectral normalization to stabilize the distribution of the post-activation values of the Transformer layers, further improving the pruning effectiveness of the proposed methodology.