Boosting Out-of-Distribution Detection with Multiple Pre-trained Models

24 Dec 2022  ·  Feng Xue, Zi He, Chuanlong Xie, Falong Tan, Zhenguo Li ·

Out-of-Distribution (OOD) detection, i.e., identifying whether an input is sampled from a novel distribution other than the training distribution, is a critical task for safely deploying machine learning systems in the open world. Recently, post hoc detection utilizing pre-trained models has shown promising performance and can be scaled to large-scale problems. This advance raises a natural question: Can we leverage the diversity of multiple pre-trained models to improve the performance of post hoc detection methods? In this work, we propose a detection enhancement method by ensembling multiple detection decisions derived from a zoo of pre-trained models. Our approach uses the p-value instead of the commonly used hard threshold and leverages a fundamental framework of multiple hypothesis testing to control the true positive rate of In-Distribution (ID) data. We focus on the usage of model zoos and provide systematic empirical comparisons with current state-of-the-art methods on various OOD detection benchmarks. The proposed ensemble scheme shows consistent improvement compared to single-model detectors and significantly outperforms the current competitive methods. Our method substantially improves the relative performance by 65.40% and 26.96% on the CIFAR10 and ImageNet benchmarks.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Out-of-Distribution Detection CIFAR-10 ZODE-KNN FPR95 3.83 # 3
AUROC 99.12 # 6
Out-of-Distribution Detection CIFAR-10 vs CIFAR-100 ZODE-KNN AUROC 97.12 # 6
FPR95 18.29 # 1

Methods