Activation Functions

StarReLU

Introduced by Yu et al. in MetaFormer Baselines for Vision

$s \cdot (\mathrm{ReLU}(x))^2 + b$

where $s \in \mathbb{R}$ and $b \in \mathbb{R}$ are shared for all channels and can be set as constants (s=0.8944, b=-0.4472) or learnable parameters.

Papers

Paper Code Results Date Stars