Skip Connections

Residual Connection

Introduced by He et al. in Deep Residual Learning for Image Recognition

Residual Connections are a type of skip-connection that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions.

Formally, denoting the desired underlying mapping as $\mathcal{H}({x})$, we let the stacked nonlinear layers fit another mapping of $\mathcal{F}({x}):=\mathcal{H}({x})-{x}$. The original mapping is recast into $\mathcal{F}({x})+{x}$.

The intuition is that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

Source: Deep Residual Learning for Image Recognition

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Language Modelling 62 7.68%
Retrieval 35 4.34%
Question Answering 30 3.72%
Large Language Model 30 3.72%
In-Context Learning 22 2.73%
Sentence 21 2.60%
Object Detection 16 1.98%
Machine Translation 15 1.86%
Translation 15 1.86%

Components


Component Type
Batch Normalization
Normalization (optional)
ReLU
Activation Functions (optional)

Categories