Normalization

Edit

General • 38 methods

Normalization layers in deep learning are used to make optimization easier by smoothing the loss surface of the network. Below you will find a continuously updating list of normalization methods.

Methods

Add a Method

Method	Year	Papers
Layer Normalization Layer Normalization	2016	24820
Batch Normalization Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	6289
Instance Normalization Instance Normalization: The Missing Ingredient for Fast Stylization	2016	571
Adaptive Instance Normalization Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization	2017	429
Local Response Normalization ImageNet Classification with Deep Convolutional Neural Networks	2012	421
Spectral Normalization Spectral Normalization for Generative Adversarial Networks	2018	220
Weight Demodulation Analyzing and Improving the Image Quality of StyleGAN	2019	182
Conditional Batch Normalization Modulating early visual processing by language	2017	145
Weight Normalization Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks	2016	88
Activation Normalization Glow: Generative Flow with Invertible 1x1 Convolutions	2018	61
Group Normalization Group Normalization	2018	55
SPADE Semantic Image Synthesis with Spatially-Adaptive Normalization	2019	37
SRN Stable Rank Normalization for Improved Generalization in Neural Networks and GANs	2019	17
Gradient Normalization Gradient Normalization for Generative Adversarial Networks	2021	15
RMSNorm Root Mean Square Layer Normalization	2019	14
MPN Is Second-order Information Helpful for Large-scale Visual Recognition?	2017	13
Weight Standardization Micro-Batch Training with Batch-Channel Normalization and Weight Standardization	2019	12
LayerScale Going deeper with Image Transformers	2021	12
Switchable Normalization Differentiable Learning-to-Normalize via Switchable Normalization	2018	9
Local Contrast Normalization	2009	8
ReZero ReZero is All You Need: Fast Convergence at Large Depth	2020	7
SyncBN Context Encoding for Semantic Segmentation	2018	5
Decorrelated Batch Normalization Decorrelated Batch Normalization	2018	4
Mixture Normalization Training Faster by Separating Modes of Variation in Batch-normalized Models	2018	4
Attentive Normalization Attentive Normalization	2019	3
Conditional Instance Normalization A Learned Representation For Artistic Style	2016	3
Online Normalization Online Normalization for Training Neural Networks	2019	3
Cosine Normalization Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks	2017	2
Mode Normalization Mode Normalization	2018	2
Filter Response Normalization Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks	2019	2
InPlace-ABN In-Place Activated BatchNorm for Memory-Optimized Training of DNNs	2017	2
BatchChannel Normalization Rethinking Normalization and Elimination Singularity in Neural Networks	2019	1
Instance-Level Meta Normalization Instance-Level Meta Normalization	2019	1
Sparse Switchable Normalization SSN: Learning Sparse Switchable Normalization via SparsestMax	2019	1
Virtual Batch Normalization Improved Techniques for Training GANs	2016	1
SaBN Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity	2021	1
EvoNorms Evolving Normalization-Activation Layers	2020	1
CInC Flow CInC Flow: Characterizable Invertible 3x3 Convolution	2021	1

Normalization Edit

Methods Add a Method

Normalization

Edit

Methods

Add a Method