An Effective Baseline for Robustness to Distributional Shift

Refraining from confidently predicting when faced with categories of inputs different from those seen during training is an important requirement for the safe deployment of deep learning systems. While simple to state, this has been a particularly challenging problem in deep learning, where models often end up making overconfident predictions in such situations. In this work we present a simple, but highly effective approach to deal with out-of-distribution detection that uses the principle of abstention: when encountering a sample from an unseen class, the desired behavior is to abstain from predicting. Our approach uses a network with an extra abstention class and is trained on a dataset that is augmented with an uncurated set that consists of a large number of out-of-distribution (OoD) samples that are assigned the label of the abstention class; the model is then trained to learn an effective discriminator between in and out-of-distribution samples. We compare this relatively simple approach against a wide variety of more complex methods that have been proposed both for out-of-distribution detection as well as uncertainty modeling in deep learning, and empirically demonstrate its effectiveness on a wide variety of of benchmarks and deep architectures for image recognition and text classification, often outperforming existing approaches by significant margins. Given the simplicity and effectiveness of this method, we propose that this approach be used as a new additional baseline for future work in this domain.

PDF Abstract

Results from the Paper

 Ranked #1 on Out-of-Distribution Detection on 20 Newsgroups (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Out-of-Distribution Detection 20 Newsgroups 2-Layered GRU AUROC 99.6 # 1
FPR95 1.78 # 1
Out-of-Distribution Detection CIFAR-10 Wide ResNet 40x2 FPR95 2.0 # 1
AUROC 99.9 # 1
Out-of-Distribution Detection CIFAR-100 Wide ResNet 40x2 FPR95 23.4 # 1
AUROC 97.7 # 1
Out-of-Distribution Detection SST 2-Layered GRU AUROC 99.7 # 1
FPR95 20.9 # 1
Out-of-Distribution Detection TREC-NEWS 2-Layered GRU AUROC 99.9 # 1
FPR95 4.7 # 1