Deep unsupervised anomaly detection
This paper proposes a novel method to detect anomalies in large datasets under a fully unsupervised setting. The key idea behind our algorithm is to learn the representation underlying normal data. To this end, we leverage the latest clustering technique suitable for handling high dimensional data. This hypothesis provides a reliable starting point for normal data selection. We train an autoencoder from the normal data subset, and iterate between hypothesizing nor-mal candidate subset based on clustering and representation learning. The reconstruction error from the learned autoencoder serves as a scoring function to assess the normality of the data. Experimental results on several public benchmark datasets show that the proposed method outperforms state-of-the-art unsupervised techniques and is comparable to semi-supervised techniques in most cases
PDFTasks
Anomaly Detection
Clustering
Representation Learning
Unsupervised Anomaly Detection
Unsupervised Anomaly Detection with Specified Settings -- 0.1% anomaly
Unsupervised Anomaly Detection with Specified Settings -- 10% anomaly
Unsupervised Anomaly Detection with Specified Settings -- 1% anomaly
Unsupervised Anomaly Detection with Specified Settings -- 20% anomaly
Unsupervised Anomaly Detection with Specified Settings -- 30% anomaly
Datasets
CIFAR-10
MNIST
Fashion-MNIST
STL-10
UCF-Crime
Cats and Dogs
Results from the Paper
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Unsupervised Anomaly Detection with Specified Settings -- 30% anomaly | ASSIRA Cat Vs Dog | Deep Unsup. | AUC-ROC | 0.740 | # 4 | |
| Unsupervised Anomaly Detection with Specified Settings -- 20% anomaly | Cats and Dogs | Deep Unsup. | AUC-ROC | 0.773 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 10% anomaly | Cats and Dogs | Deep Unsup. | AUC-ROC | 0.862 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 0.1% anomaly | Cats and Dogs | Deep Unsup. | AUC-ROC | 0.545 | # 6 | |
| Unsupervised Anomaly Detection with Specified Settings -- 1% anomaly | Cats and Dogs | Deep Unsup. | AUC-ROC | 0.862 | # 4 | |
| Unsupervised Anomaly Detection with Specified Settings -- 20% anomaly | cifar10 | Deep Unsup. | AUC-ROC | 0.702 | # 6 | |
| Unsupervised Anomaly Detection with Specified Settings -- 10% anomaly | CIFAR-10 | Deep Unsup. | AUC-ROC | 0.847 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 0.1% anomaly | CIFAR-10 | Deep Unsup. | AUC-ROC | 0.841 | # 4 | |
| Unsupervised Anomaly Detection with Specified Settings -- 1% anomaly | CIFAR-10 | Deep Unsup. | AUC-ROC | 0.847 | # 4 | |
| Unsupervised Anomaly Detection with Specified Settings -- 30% anomaly | CIFAR-10 | Deep Unsup. | AUC-ROC | 0.689 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 10% anomaly | Fashion-MNIST | Deep Unsup. | AUC-ROC | 0.878 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 0.1% anomaly | Fashion-MNIST | Deep Unsup. | AUC-ROC | 0.765 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 20% anomaly | Fashion-MNIST | Deep Unsup. | AUC-ROC | 0.884 | # 2 | |
| Unsupervised Anomaly Detection with Specified Settings -- 1% anomaly | Fashion-MNIST | Deep Unsup. | AUC-ROC | 0.868 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 30% anomaly | Fashion-MNIST | Deep Unsup. | AUC-ROC | 0.856 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 1% anomaly | MNIST | Deep Unsup. | AUC-ROC | 0.891 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 30% anomaly | MNIST | Deep Unsup. | AUC-ROC | 0.835 | # 2 | |
| Unsupervised Anomaly Detection with Specified Settings -- 10% anomaly | MNIST | Deep Unsup. | AUC-ROC | 0.847 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 0.1% anomaly | MNIST | Deep Unsup. | AUC-ROC | 0.525 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 20% anomaly | MNIST | Deep Unsup. | AUC-ROC | 0.779 | # 4 | |
| Unsupervised Anomaly Detection with Specified Settings -- 1% anomaly | STL-10 | Deep Unsup. | AUC-ROC | 0.956 | # 3 | |
| Unsupervised Anomaly Detection with Specified Settings -- 0.1% anomaly | STL-10 | Deep Unsup. | AUC-ROC | 0.384 | # 6 | |
| Unsupervised Anomaly Detection with Specified Settings -- 20% anomaly | STL-10 | Deep Unsup. | AUC-ROC | 0.869 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 30% anomaly | STL-10 | Deep Unsup. | AUC-ROC | 0.866 | # 5 | |
| Unsupervised Anomaly Detection with Specified Settings -- 10% anomaly | STL-10 | Deep Unsup. | AUC-ROC | 0.906 | # 4 |
