Kaiming Initialization

Introduced by He et al. in Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming Initialization, or He Initialization, is an initialization method for neural networks that takes into account the non-linearity of activation functions, such as ReLU activations.

A proper initialization method should avoid reducing or magnifying the magnitudes of input signals exponentially. Using a derivation they work out that the condition to stop this happening is:

$$\frac{1}{2}n_{l}\text{Var}\left[w_{l}\right] = 1 $$

This implies an initialization scheme of:

$$ w_{l} \sim \mathcal{N}\left(0, 2/n_{l}\right)$$

That is, a zero-centered Gaussian with standard deviation of $\sqrt{2/{n}_{l}}$ (variance shown in equation above). Biases are initialized at $0$.

Source: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Read Paper See Code

Papers

Paper	Code	Results	Date	Stars

Tasks

Task	Papers	Share
Image Classification	53	7.99%
Self-Supervised Learning	49	7.39%
Semantic Segmentation	34	5.13%
Classification	33	4.98%
Object Detection	15	2.26%
Image Segmentation	13	1.96%
Quantization	11	1.66%
Decoder	10	1.51%
Speaker Verification	7	1.06%

Usage Over Time

This feature is experimental; we are continuously improving our matching algorithm.

Components

Component	Type	Add Remove
🤖 No Components Found	You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories

Add Remove

Initialization