AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection

30 Jun 2022  ·  Marius Dragoi, Elena Burceanu, Emanuela Haller, Andrei Manolache, Florin Brad ·

Analyzing the distribution shift of data is a growing research direction in nowadays Machine Learning (ML), leading to emerging new benchmarks that focus on providing a suitable scenario for studying the generalization properties of ML models. The existing benchmarks are focused on supervised learning, and to the best of our knowledge, there is none for unsupervised learning. Therefore, we introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. This type of data meets the premise of shifting the input distribution: it covers a large time span ($10$ years), with naturally occurring changes over time (eg users modifying their behavior patterns, and software updates). We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. Next, we propose AnoShift, a protocol splitting the data in IID, NEAR, and FAR testing splits. We validate the performance degradation over time with diverse models, ranging from classical approaches to deep learning. Finally, we show that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical training which assumes independent and identically distributed data (on average, by up to $3\%$ for our approach). Dataset and code are available at https://github.com/bit-ml/AnoShift/.

PDF Abstract

Datasets


Introduced in the Paper:

AnoShift
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Unsupervised Anomaly Detection AnoShift COPOD ROC-AUC IID 85.62 # 4
ROC-AUC NEAR 54.24 # 7
ROC-AUC FAR 50.42 # 1
Unsupervised Anomaly Detection AnoShift Internal Contrastive Learning ROC-AUC IID 84.86 # 5
ROC-AUC NEAR 52.26 # 8
ROC-AUC FAR 22.45 # 10
Unsupervised Anomaly Detection AnoShift LUNAR ROC-AUC IID 85.75 # 3
ROC-AUC NEAR 49.03 # 9
ROC-AUC FAR 28.19 # 7
Unsupervised Anomaly Detection AnoShift AE for anomalies ROC-AUC IID 81 # 8
ROC-AUC NEAR 44.06 # 11
ROC-AUC FAR 19.96 # 11
Unsupervised Anomaly Detection AnoShift deepSVDD ROC-AUC IID 73.43 # 10
ROC-AUC NEAR 69.61 # 5
ROC-AUC FAR 31.81 # 6
Unsupervised Anomaly Detection AnoShift SO-GAAL ROC-AUC IID 50.48 # 11
ROC-AUC NEAR 54.55 # 6
ROC-AUC FAR 49.35 # 3
Unsupervised Anomaly Detection AnoShift ECOD Li et al. (2022) ROC-AUC IID 84.76 # 6
ROC-AUC NEAR 44.87 # 10
ROC-AUC FAR 49.19 # 4
Unsupervised Anomaly Detection AnoShift IsoForest ROC-AUC IID 86.09 # 2
ROC-AUC NEAR 75.26 # 3
ROC-AUC FAR 27.16 # 9
Unsupervised Anomaly Detection AnoShift OC-SVM ROC-AUC IID 76.86 # 9
ROC-AUC NEAR 71.43 # 4
ROC-AUC FAR 49.57 # 2
Anomaly Detection AnoShift BERT ROC-AUC IID 84.54 # 1
Unsupervised Anomaly Detection AnoShift BERT ROC-AUC IID 84.54 # 7
ROC-AUC NEAR 86.05 # 1
ROC-AUC FAR 28.15 # 8
Unsupervised Anomaly Detection AnoShift LOF ROC-AUC IID 91.5 # 1
ROC-AUC NEAR 79.29 # 2
ROC-AUC FAR 34.96 # 5

Methods