Normalization

Edit

General • 36 methods

Normalization layers in deep learning are used to make optimization easier by smoothing the loss surface of the network. Below you will find a continuously updating list of normalization methods.

Methods

Add a Method

Method	Year	Papers
Layer Normalization Layer Normalization	2016	8462
Batch Normalization Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	4127
Local Response Normalization ImageNet Classification with Deep Convolutional Neural Networks	2012	377
Instance Normalization Instance Normalization: The Missing Ingredient for Fast Stylization	2016	344
Adaptive Instance Normalization Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization	2017	211
Spectral Normalization Spectral Normalization for Generative Adversarial Networks	2018	137
Weight Demodulation Analyzing and Improving the Image Quality of StyleGAN	2019	112
Conditional Batch Normalization Modulating early visual processing by language	2017	82
Weight Normalization Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks	2016	70
Group Normalization Group Normalization	2018	43
Activation Normalization Glow: Generative Flow with Invertible 1x1 Convolutions	2018	33
SPADE Semantic Image Synthesis with Spatially-Adaptive Normalization	2019	22
LayerScale Going deeper with Image Transformers	2021	9
Weight Standardization Micro-Batch Training with Batch-Channel Normalization and Weight Standardization	2019	8
Switchable Normalization Differentiable Learning-to-Normalize via Switchable Normalization	2018	8
MPN Is Second-order Information Helpful for Large-scale Visual Recognition?	2017	8
Local Contrast Normalization	2009	7
SRN Stable Rank Normalization for Improved Generalization in Neural Networks and GANs	2019	7
SyncBN Context Encoding for Semantic Segmentation	2018	5
Attentive Normalization Attentive Normalization	2019	3
Decorrelated Batch Normalization Decorrelated Batch Normalization	2018	3
Conditional Instance Normalization A Learned Representation For Artistic Style	2016	3
Online Normalization Online Normalization for Training Neural Networks	2019	3
Gradient Normalization Gradient Normalization for Generative Adversarial Networks	2021	3
ReZero ReZero is All You Need: Fast Convergence at Large Depth	2020	3
Cosine Normalization Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks	2017	2
Filter Response Normalization Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks	2019	2
InPlace-ABN In-Place Activated BatchNorm for Memory-Optimized Training of DNNs	2017	2
BatchChannel Normalization Rethinking Normalization and Elimination Singularity in Neural Networks	2019	1
Instance-Level Meta Normalization Instance-Level Meta Normalization	2019	1
Mixture Normalization Training Faster by Separating Modes of Variation in Batch-normalized Models	2018	1
Mode Normalization Mode Normalization	2018	1
Sparse Switchable Normalization SSN: Learning Sparse Switchable Normalization via SparsestMax	2019	1
Virtual Batch Normalization Improved Techniques for Training GANs	2016	1
SaBN Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity	2021	1
EvoNorms Evolving Normalization-Activation Layers	2020	1

Normalization Edit

Methods Add a Method

Normalization

Edit

Methods

Add a Method