For most state-of-the-art architectures, Rectified Linear Unit (ReLU) becomes a standard component accompanied with each layer. Although ReLU can ease the network training to an extent, the character of blocking negative values may suppress the propagation of useful information and leads to the difficulty of optimizing very deep Convolutional Neural Networks (CNNs). Moreover, stacking layers with nonlinear activations is hard to approximate the intrinsic linear transformations between feature representations.
|Task||Dataset||Model||Metric name||Metric value||Global rank||Compare|
|Image Classification||SVHN||EraseReLU||Percentage error||1.54||# 2|