SAU: Smooth activation function using convolution with approximate identities
Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with ShuffleNet V2 (2.0x) model on CIFAR100 dataset.
PDF AbstractTasks
Datasets
CIFAR-10
CIFAR-100
Cityscapes
SVHN
Fashion-MNIST
ssd
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
1x1 Convolution •
Average Pooling •
Batch Normalization •
Channel Shuffle •
Convolution •
Dense Connections •
Depthwise Convolution •
Global Average Pooling •
Grouped Convolution •
Groupwise Point Convolution •
Leaky ReLU •
Max Pooling •
Pointwise Convolution •
ReLU •
Residual Connection •
ShuffleNet •
ShuffleNet Block •
ShuffleNet v2 •
ShuffleNet V2 Block •
ShuffleNet V2 Downsampling Block •
Softmax
