TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK	REMOVE
Image Classification	CIFAR-10	RReLU	Percentage correct	88.8	# 195
Image Classification	CIFAR-100	RReLU	Percentage correct	59.8	# 190

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/empirical-evaluation-of-rectified-activations/image-classification-on-cifar-100)](https://paperswithcode.com/sota/image-classification-on-cifar-100?p=empirical-evaluation-of-rectified-activations)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/empirical-evaluation-of-rectified-activations/image-classification-on-cifar-10)](https://paperswithcode.com/sota/image-classification-on-cifar-10?p=empirical-evaluation-of-rectified-activations)`

Empirical Evaluation of Rectified Activations in Convolutional Network

5 May 2015 · Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li ·

In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.

PDF Abstract