Normalization

Edit

General • 31 methods

Normalization layers in deep learning are used to make optimization easier by smoothing the loss surface of the network. Below you will find a continuously updating list of normalization methods.

Methods

Add a Method

Method	Year	Papers
Layer Normalization Layer Normalization	2016	4416
Batch Normalization Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	2930
Local Response Normalization ImageNet Classification with Deep Convolutional Neural Networks	2012	358
Instance Normalization Instance Normalization: The Missing Ingredient for Fast Stylization	2016	244
Adaptive Instance Normalization Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization	2017	105
Spectral Normalization Spectral Normalization for Generative Adversarial Networks	2018	100
Conditional Batch Normalization Modulating early visual processing by language	2017	56
Weight Normalization Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks	2016	54
Weight Demodulation Analyzing and Improving the Image Quality of StyleGAN	2019	45
Group Normalization Group Normalization	2018	36
Activation Normalization Glow: Generative Flow with Invertible 1x1 Convolutions	2018	24
Weight Standardization Micro-Batch Training with Batch-Channel Normalization and Weight Standardization	2019	7
Switchable Normalization Differentiable Learning-to-Normalize via Switchable Normalization	2018	7
MPN Is Second-order Information Helpful for Large-scale Visual Recognition?	2017	7
Local Contrast Normalization	2009	4
SyncBN Context Encoding for Semantic Segmentation	2018	4
Conditional Instance Normalization A Learned Representation For Artistic Style	2016	3
Cosine Normalization Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks	2017	2
Attentive Normalization Attentive Normalization	2019	2
Decorrelated Batch Normalization Decorrelated Batch Normalization	2018	2
Filter Response Normalization Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks	2019	2
InPlace-ABN In-Place Activated BatchNorm for Memory-Optimized Training of DNNs	2017	2
SRN Stable Rank Normalization for Improved Generalization in Neural Networks and GANs	2019	2
Online Normalization Online Normalization for Training Neural Networks	2019	2
BatchChannel Normalization Rethinking Normalization and Elimination Singularity in Neural Networks	2019	1
Instance-Level Meta Normalization Instance-Level Meta Normalization	2019	1
Mixture Normalization Training Faster by Separating Modes of Variation in Batch-normalized Models	2018	1
Mode Normalization Mode Normalization	2018	1
Sparse Switchable Normalization SSN: Learning Sparse Switchable Normalization via SparsestMax	2019	1
Virtual Batch Normalization Improved Techniques for Training GANs	2016	1
SaBN Sandwich Batch Normalization	2021	1

Normalization Edit

Methods Add a Method

Normalization

Edit

Methods

Add a Method