PATS: A New Neural Network Activation Function with Parameter

Activation function is crucial to the recent successes of deep neural networks. In this paper, we propose a new activation function with parameters, named PATS. Specifically, PATS is a non-monotonic function which combines arctangent function and sigmoid function. In the process of network model training, the parameter of PATS is a random number from the uniform distribution, which improves the flexibility of network model and reduces the risk of over fitting. In addition, PATS can be widely used in existing deep network models. We use several classic deep network models to test the performance of PATS. The experimental results on CIFAR-10 and CIFAR-100 datasets show that compared with other activation functions, PATS has better performance in improving the learning ability and robustness of deep network model.