Normalization
Normalization layers in deep learning are used to make optimization easier by smoothing the loss surface of the network. Below you will find a continuously updating list of normalization methods.
| METHOD | YEAR | PAPERS | |
|---|---|---|---|
Layer Normalization
|
2016 | 2285 | |
Batch Normalization
|
2015 | 1997 | |
Local Response Normalization
|
2012 | 368 | |
Instance Normalization
|
2016 | 168 | |
Spectral Normalization
|
2018 | 63 | |
Weight Normalization
|
2016 | 47 | |
Adaptive Instance Normalization
|
2017 | 46 | |
Conditional Batch Normalization
|
2017 | 35 | |
Group Normalization
|
2018 | 25 | |
Activation Normalization
|
2018 | 15 | |
Weight Demodulation
|
2019 | 8 | |
Switchable Normalization
|
2018 | 6 | |
Weight Standardization
|
2019 | 3 | |
Local Contrast Normalization
|
2009 | 3 | |
Conditional Instance Normalization
|
2016 | 3 | |
SyncBN
|
2018 | 3 | |
Attentive Normalization
|
2019 | 2 | |
Decorrelated Batch Normalization
|
2018 | 2 | |
InPlace-ABN
|
2017 | 2 | |
Cosine Normalization
|
2017 | 1 | |
BatchChannel Normalization
|
2019 | 1 | |
Instance-Level Meta Normalization
|
2019 | 1 | |
Mixture Normalization
|
2018 | 1 | |
Mode Normalization
|
2018 | 1 | |
Sparse Switchable Normalization
|
2019 | 1 | |
Filter Response Normalization
|
2019 | 1 | |
Virtual Batch Normalization
|
2016 | 1 | |
SRN
|
2019 | 1 | |
Online Normalization
|
2019 | 1 |