BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning

5 Oct 2020  ·  Zhijie Deng, Hao Zhang, Xiao Yang, Yinpeng Dong, Jun Zhu ·

Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, due to persistent concerns on their scalability, accessibility, and reliability. In this work, we aim to relieve these concerns by developing the BayesAdapter framework for learning variational BNNs. In particular, we propose to adapt the pre-trained deterministic NNs to be BNNs via cost-effective Bayesian fine-tuning. To make BayesAdapter more practical, we technically contribute 1) a modularized, user-friendly implementation for the learning of variational BNNs under two representative variational distributions, 2) a generally applicable strategy for reducing the gradient variance in stochastic variational inference, 3) an explanation for the unreliability issue of BNNs' uncertainty estimates, and a corresponding prescription. Through extensive experiments on diverse benchmarks, we show that BayesAdapter can consistently induce posteriors with higher quality than the from-scratch variational inference and other competitive baselines, especially in large-scale settings, yet significantly reducing training overheads.

PDF Abstract

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods


No methods listed for this paper. Add relevant methods here