MVTec AD is a dataset for benchmarking anomaly detection methods with a focus on industrial inspection. It contains over 5000 high-resolution images divided into fifteen different object and texture categories. Each category comprises a set of defect-free training images and a test set of images with various kinds of defects as well as images without defects.

There are two common metrics: Detection AUROC and Segmentation (or pixelwise) AUROC

Detection (or, classification) methods output single float (anomaly score) per input test image.

Segmentation methods output anomaly probability for each pixel. "To assess segmentation performance, we evaluate the relative per-region overlap of the segmentation with the ground truth. To get an additional performance measure that is independent of the determined threshold, we compute the area under the receiver operating characteristic curve (ROC AUC). We define the true positive rate as the percentage of pixels that were correctly classified as anomalous" [1] Later segmentation metric was improved to balance regions with small and large area, see PRO-AUC and other in [2]

[1] Paul Bergmann et al, "MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection" [2] Bergmann, P., Batzner, K., Fauser, M. et al. The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. Int J Comput Vis (2021). https://doi.org/10.1007/s11263-020-01400-4