Activation Functions

Activation functions are functions that we apply in neural networks after (typically) applying an affine transformation combining weights and input features. They are typically non-linear functions. The rectified linear unit, or ReLU, has been the most popular in the past decade, although the choice is architecture dependent and many alternatives have emerged in recent years. In this section, you will find a constantly updating list of activation functions.

METHOD YEAR PAPERS
ReLU
2000 4364
Sigmoid Activation
2000 3009
Tanh Activation
2000 2850
GELU
2016 1232
Leaky ReLU
2014 325
Swish
2017 54
PReLU
2015 43
Maxout
2013 31
Softplus
2000 18
ELU
2015 16
GLU
2016 14
SELU
2017 13
ReLU6
2017 10
Softsign Activation
2000 9
Hard Swish
2019 9
Mish
2019 5
Hard Sigmoid
2015 4
CReLU
2016 4
RReLU
2015 3
KAF
2017 3
SReLU
2015 2
modReLU
2015 2
Hermite Activation
2000 2
SiLU
2017 1
PELU
2016 1
ELiSH
2018 1
HardELiSH
2018 1
SERLU
2018 1
Lecun's Tanh
1998 0
Hardtanh Activation
2000 0