Offline Deep models calibration with bayesian neural networks

We apply Bayesian Neural Networks to improve calibration of state-of-the-art deep neural networks. We show that, even with the most basic amortized approximate posterior distribution, and fast fully connected neural network for the likelihood, the Bayesian framework clearly outperforms other simple maximum likelihood based solutions that have recently shown very good performance, as temperature scaling. As an example, we reduce the Expected Calibration Error (ECE) from 0.52 to 0.24 on CIFAR-10 and from 4.28 to 2.456 on CIFAR-100 on two Wide ResNet with 96.13% and 80.39% accuracy respectively, which are among the best results published for this task. We demonstrate our robustness and performance with experiments on a wide set of state-of-the-art computer vision models. Moreover, our approach acts off-line, and thus can be applied to any probabilistic model regardless of the limitations that the model may present during training. This make it suitable to calibrate systems that make use of pre-trained deep neural networks that are expensive to train for a specific task, or to directly train a calibrated deep convolutional model with Monte Carlo Dropout approximations, among others. However, our method is still complementary with any Bayesian Neural Network for further improvement.