Regularization

Edit

General • 56 methods

Regularization strategies are designed to reduce the test error of a machine learning algorithm, possibly at the expense of training error. Many different forms of regularization exist in the field of deep learning. Below you can find a constantly updating list of regularization strategies.

Subcategories

1 Parameter Norm Penalties

Methods

Add a Method

Method	Year	Papers
Dropout Dropout: A Simple Way to Prevent Neural Networks from Overfitting	2014	15741
Label Smoothing	1985	7191
Weight Decay	1943	6335
Attention Dropout	2018	6296
Entropy Regularization Asynchronous Methods for Deep Reinforcement Learning	2016	424
R1 Regularization Which Training Methods for GANs do actually Converge?	2018	419
Early Stopping	1995	332
Stochastic Depth Deep Networks with Stochastic Depth	2016	269
Path Length Regularization Analyzing and Improving the Image Quality of StyleGAN	2019	161
Off-Diagonal Orthogonal Regularization Large Scale GAN Training for High Fidelity Natural Image Synthesis	2018	131
Variational Dropout A Theoretically Grounded Application of Dropout in Recurrent Neural Networks	2015	125
DropBlock DropBlock: A regularization method for convolutional networks	2018	115
DropConnect Regularization of Neural Networks using DropConnect	2013	76
Target Policy Smoothing Addressing Function Approximation Error in Actor-Critic Methods	2018	70
L1 Regularization	1986	62
Embedding Dropout A Theoretically Grounded Application of Dropout in Recurrent Neural Networks	2015	57
ALS Efficient Model for Image Classification With Regularization Tricks	2020	51
Activation Regularization Revisiting Activation Regularization for Language RNNs	2017	48
Temporal Activation Regularization Revisiting Activation Regularization for Language RNNs	2017	48
PGM	2000	47
SpatialDropout Efficient Object Localization Using Convolutional Networks	2014	25
Zoneout Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations	2016	25
LCC Regularisation of Neural Networks by Enforcing Lipschitz Continuity	2018	23
DropPath FractalNet: Ultra-Deep Neural Networks without Residuals	2016	22
Manifold Mixup Manifold Mixup: Better Representations by Interpolating Hidden States	2018	20
Orthogonal Regularization Neural Photo Editing with Introspective Adversarial Networks	2016	19
GAN Feature Matching Improved Techniques for Training GANs	2016	15
Concrete Dropout Concrete Dropout	2017	14
SRN Stable Rank Normalization for Improved Generalization in Neural Networks and GANs	2019	12
SCN Suppressing Uncertainties for Large-Scale Facial Expression Recognition	2020	12
LayerScale Going deeper with Image Transformers	2021	10
Euclidean Norm Regularization Deep Compressed Sensing	2019	9
Auxiliary Batch Normalization Adversarial Examples Improve Image Recognition	2019	9
Adaptive Dropout Adaptive dropout for training deep neural networks	2013	7
Shake-Shake Regularization Shake-Shake regularization	2017	5
SVD Parameterization Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization	2018	4
FIERCE Preserving Fine-Grain Feature Information in Classification via Entropic Regularization	2022	3
ShakeDrop ShakeDrop Regularization for Deep Residual Learning	2018	3
GradDrop Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout	2020	3
Recurrent Dropout Recurrent Dropout without Memory Loss	2016	3
GMVAE Regularizing Transformers With Deep Probabilistic Layers	2021	3
Batch Nuclear-norm Maximization Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations	2020	3
LayerDrop Reducing Transformer Depth on Demand with Structured Dropout	2019	3
Discriminative Regularization Discriminative Regularization for Generative Models	2016	2
Targeted Dropout Learning Sparse Networks Using Targeted Dropout	2019	2
Fraternal Dropout Fraternal Dropout	2017	2
ScheduledDropPath Learning Transferable Architectures for Scalable Image Recognition	2017	2
DropPathway Audiovisual SlowFast Networks for Video Recognition	2020	1
AutoDropout AutoDropout: Learning Dropout Patterns to Regularize Deep Networks	2021	1
Spectral Dropout Regularization of Deep Neural Networks with Spectral Dropout	2017	1
Early Dropout Dropout Reduces Underfitting	2023	1
Weights Reset The Weights Reset Technique for Deep Neural Networks Implicit Regularization	2023	1
STTP Spectral Tensor Train Parameterization of Deep Learning Layers	2021	1
rnnDrop	2015	0

Regularization Edit

Methods Add a Method

Regularization

Edit

Methods

Add a Method