Temporal Activation Regularization

Introduced by Merity et al. in Revisiting Activation Regularization for Language RNNs

Temporal Activation Regularization (TAR) is a type of slowness regularization for RNNs that penalizes differences between states that have been explored in the past. Formally we minimize:

$$\beta{L_{2}}\left(h_{t} - h_{t+1}\right)$$

where $L_{2}$ is the $L_{2}$ norm, $h_{t}$ is the output of the RNN at timestep $t$, and $\beta$ is a scaling coefficient.

Source: Revisiting Activation Regularization for Language RNNs


Paper Code Results Date Stars


Task Papers Share
Language Modelling 20 17.39%
General Classification 14 12.17%
Text Classification 13 11.30%
Classification 8 6.96%
Sentiment Analysis 8 6.96%
Language Identification 4 3.48%
Translation 4 3.48%
Test 4 3.48%
Hate Speech Detection 3 2.61%


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign