My highly-acclaimed master's thesis project: Industrial Anomaly Detection Benchmarking Engine

1. Architecture

As shown in the figure, the IADBE (Industrial Anomaly Detection Benchmark Engine) system consists of three main components: the IADBE, the IADBE server, and the IADBE backend.The IADBE is the core of the system, and the API and CLI are the gateways. The datasets, models and metrics are the most important parts of the system. The models are based on the open source Anomalib and YOLOv8. There are three main entry points to the system: training, testing and prediction.

In the IADBE server, we used SpringBoot and Kotlin frameworks.The IADBE server gets the training and test result files from IADBE and then processes them, including logging and CSV processing. For the IADBE front-end, we use the Angular framework. The front-end sends HTTP requests to the server, which then sends JSON data back to the front-end. The front-end provides stylish and functional visualizations.
To simplify the deployment process, we use Docker Compose to put together IADBE, IADBE Server, and IADBE Front End images and run them together. We use GitHub for version control. We publish all relevant software, including IADBE, IADBE Server, and IADBE Frontend, on GitHub for replication and validation. We have uploaded our custom dataset and pre-trained models to Huggingface. we tested the cross-platform functionality on a range of operating systems, including Windows 11 and 10, and Ubuntu 22 and 20. the results show that these operating systems have no problems running the IADBE system.

2. Introduction

It is worth noting that previous research accompanying open source projects has often been time-consuming due to environment deployment issues, and in some cases projects have failed to run due to version conflict issues. However, in this project, the solution will be deployed using Docker or Conda. The main goal of this project is to provide researchers with an out-of-the-box industrial anomaly detection platform. The platform, which is mainly based on Anomalib, Ultralytics, should be able to reproduce and identify previous studies, be as bug-free as possible and be easy to deploy.

3. Installation

Tested on Linux (Ubuntu22/20), Windows (Win11/10) ✅

IADBE offers two ways to install libraries: Conda and Docker. If you want to modify dependency packages and their versions and work in development mode, use Conda. If you want to replicate our environment exactly (python, torrent......) , please use Docker. ⚠️ assumes that you have the nvidia driver and CUDA installed. Otherwise, you can use the CPU for training.

3.1 Conda Installation

Installing dependencies from Conda

# Use of virtual environment is highly recommended
# Using conda
conda create -n IADBE python=3.10
conda activate IADBE

# To avoid anomalib install bug
pip install pip==24.0

# Install anomalib
pip install anomalib==1.1.0

# Install the full package, this will install Anomalib CLI. Anomalib CLI is a command line interface for training, testing.
anomalib install

# Install ultralytics
pip install ultralytics

# Or using your favorite virtual environment
# ...

3.2 Docker Installation

Installing dependencies from Docker

# Clone the repository and install in editable mode
git clone /cjy513203427/
cd IADBE

# Build docker image
docker build --no-cache -t iadbe .
# Run docker container
docker run --gpus all -it --shm-size="8g" --name iadbe --rm iadbe bash

You can use it as a virtual machine with the same commands for training, testing, and reasoning, or you can set up docker env as an external environment.

4. Data sets

4.1 Standard data sets

IADBE can automatically download standard datasets (MVTec, MVTec3D, Btech, VisA, Kolektor) via API/CLI. If you have any problems downloading datasets, you can also download them directly from their official website.

4.2 Customized data sets

We uploaded a custom dataset to theIADBE_Custom_Dataset. It contains datasets in anomalib and YOLO formats. You can import them using the Huggingface method, or clone them from GitHub and download them locally.

4.2.1 Using huggingface

from datasets import load_dataset
ds = load_dataset("gt111lk/IADBE_Custom_Dataset")

4.2.2 Git Clone

git clone /datasets/gt111lk/IADBE_Custom_Dataset

5. Training and testing

IADBE supports both API-based and CLI-based training. the API is more flexible and allows for more customization, while CLI-based training uses a command-line interface that may be easier for those who want to get started with IADBE quickly.

5.1 Training and testing using APIs

The train_test_mvtec_xxx.py file looks like this. Run this file using the IDE or python train_test_mvtec_xxx.py to start training with the entire MVTec dataset.

import logging
from anomalib import TaskType
from  import MVTec
from  import Engine
from  import Padim

# configure logger
(level=, format='%(asctime)s - %(levelname)s - %(message)s')
logger = (__name__)

datasets = ['screw', 'pill', 'capsule', 'carpet', 'grid', 'tile', 'wood', 'zipper', 'cable', 'toothbrush', 'transistor',
            'metal_nut', 'bottle', 'hazelnut', 'leather']

for dataset in datasets:
    (f"================== Processing dataset: {dataset} ==================")
    model = Padim()
    datamodule = MVTec(category=dataset, num_workers=0, train_batch_size=256,
                       eval_batch_size=256)
    engine = Engine(pixel_metrics=["AUROC", "PRO"], image_metrics=["AUROC", "PRO"], task=)

    (f"================== Start training for dataset: {dataset} ==================")
    (model=model, datamodule=datamodule)

    (f"================== Start testing for dataset: {dataset} ==================")
    test_results = (
        model=model,
        datamodule=datamodule,
        ckpt_path=.checkpoint_callback.best_model_path,
    )

5.2 Using the CLI for training and testing

The train_test_mvtec_xxx.sh file looks like this. Run this file with bash train_test_mvtec_xxx.sh to start training with the entire MVTec dataset.

#!/bin/bash

datasets=('screw' 'pill' 'capsule' 'carpet' 'grid' 'tile' 'wood' 'zipper' 'cable' 'toothbrush' 'transistor' 'metal_nut' 'bottle' 'hazelnut' 'leather')
config_file="./configs/models/"

for dataset in "${datasets[@]}"
do
    command="anomalib train --data  -- $dataset --config $config_file"
    echo "Running command: $command"
    # Excute command
    $command
done

For an in-depth look at the Anomalib CLI, you can search theTraining via CLI from Training

6. Reasoning

Anomalib contains several inference scripts, including Torch, Lightning, Gradio, and OpenVINO reasoners, that perform inference using trained/exported models. Here we show an example of inference using the Lightning inductor. If you want to test our pre-trained models without training them, you can find them in theHuggingfaceIADBE_Models Find it on.

6.1 Reasoning with the API

The following example demonstrates how to perform Lightning inference by loading a model from a checkpoint file.

# Assuming the datamodule, custom_model and engine is initialized from the previous step,
# a prediction via a checkpoint file can be performed as follows:
predictions = (
    datamodule=datamodule,
    model=model,
    ckpt_path="path/to/",
)

6.2 Reasoning with the CLI

# To get help about the arguments, run:
anomalib predict -h

# Predict by using the default values.
anomalib predict --model  \
                 --data  \
                 --ckpt_path <path/to/custom_model.ckpt>

# Predict by overriding arguments.
anomalib predict --model  \
                 --data  \
                 --ckpt_path <path/to/custom_model.ckpt>
                 --return_predictions

# Predict by using a config file.
anomalib predict --config <path/to/config> --return_predictions

7. Customized data set models

IADBE can help you train and reason on customized datasets. The default dataset format is based on Anomalib, but you can also use YOLO. just check out theyolo_custom_dataset_setting.

The first thing to do is to import your own dataset into the project and create a custom data profile.

Normal image only configuration

class_path: 
init_args:
  name: "custom_dataset"
  root: "datasets/Custom_Dataset/hazelnut"
  normal_dir: "train/good"
  abnormal_dir: "test/crack"
  mask_dir: null
  normal_split_ratio: 0.2
  test_split_mode: synthetic

Configuration with normal and abnormal images

class_path: 
init_args:
  name: "custom_dataset"
  root: "datasets/Custom_Dataset/chest_xray"
  normal_dir: "train/good"
  abnormal_dir: "test/crack"
  normal_test_dir: "test/good"
  normal_split_ratio: 0
  extensions: [".png"]
  image_size: [256, 256]
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 8
  task: classification
  train_transform: null
  eval_transform: null
  test_split_mode: synthetic
  test_split_ratio: 0.2
  val_split_mode: same_as_test
  val_split_ratio: 0.5
  seed: null

Next, you need to select a model and then train it using a custom dataset.

anomalib train --data <path/to/custom_dataset.yaml> --model

Finally, you can run inference using a pre-trained model to get predictive results.

anomalib predict --model  \
                 --data <path/to/custom_dataset/custom_image.png> \
                 --ckpt_path <path/to/custom_dataset.ckpt>

8. Benchmarking results

These are some of the results from the benchmarks. You can find more details in my paper in the GitHub container folderpapers/ . These results are from the original logs, which you can find in thelogs/rawlogs Found under. These models were evaluated on Ubuntu 22.04 and RTX 3090.

	Screw	Pill	Capsule	Carpet	Grid	Tile	Wood	Zipper	Cable	Toothbrush	Transistor	Metal Nut	Bottle	Hazelnut	Leather	Average	Source
CFA	87.68	93.10	48.17	50.00	84.04	37.50	62.50	40.34	61.96	98.06	45.00	39.78	50.00	60.00	50.00	60.54	99.30
CFLOW	81.57	92.91	95.45	97.27	88.47	100.00	99.39	97.64	94.15	94.17	94.88	100.00	100.00	99.68	100.00	95.71	96.31
CSFLOW	41.58	38.09	60.77	99.16	78.36	90.44	95.75	88.93	67.31	44.72	50.54	74.46	89.92	71.02	100.00	72.74	93.50
DFKDE	69.17	64.18	72.04	68.42	46.69	92.50	81.32	88.26	68.67	78.89	81.12	76.20	92.94	77.57	77.79	75.72	77.40
DFM	77.79	96.89	92.98	90.93	65.41	98.67	97.81	97.51	94.27	96.39	94.79	91.72	100.00	96.82	100.00	92.80	94.30
DRAEM	30.01	74.55	77.02	72.47	80.45	86.80	95.96	78.90	63.98	70.42	90.21	93.55	98.10	77.50	86.65	78.44	98.00
DSR	56.67	70.16	72.72	43.82	97.08	80.70	90.75	78.97	76.96	96.94	91.04	81.04	86.59	81.64	83.97	79.27	98.20
FASTFLOW	65.85	76.60	69.64	93.38	96.32	93.36	98.16	72.69	67.62	72.50	89.62	81.33	99.68	79.86	99.90	83.77	99.40
FRE	58.11	83.82	76.59	94.18	64.75	98.99	98.16	93.91	85.53	95.00	90.92	84.12	99.37	92.29	99.93	87.71	98.40
GANomaly	32.79	59.96	26.57	21.71	57.23	54.42	60.88	41.05	52.47	49.17	33.46	26.30	47.78	53.86	36.68	43.62	42.10
PaDiM	78.95	79.95	86.48	97.99	87.47	94.55	97.46	77.46	85.96	82.50	94.54	98.34	99.52	88.32	100.00	89.97	96.70
PatchCore	98.11	94.76	97.85	99.12	98.08	98.81	98.77	99.21	99.10	100.00	100.00	99.80	100.00	100.00	100.00	98.91	98.10
RD	98.03	97.63	97.93	99.36	95.49	100.00	99.39	97.16	95.45	91.39	97.87	100.00	100.00	100.00	100.00	97.98	98.50
RKDE	50.58	68.77	51.93	-	75.36	67.72	62.54	75.37	85.83	77.17	65.00	90.63	85.07	100.00	100.00	75.43	-
STFPM	77.62	40.78	60.59	98.88	59.23	97.08	98.95	75.39	91.34	47.50	61.88	40.22	43.97	96.50	100.00	72.66	97.00
UFLOW	49.19	94.84	56.72	100.00	99.33	99.39	95.09	89.73	62.67	64.17	81.46	55.77	99.21	90.39	100.00	82.53	98.74

These are the baseline results regarding Image AUROC. The last column shows the results declared by the authors of the source papers.There is a significant difference between the established results of CFA (99.30) and the average results derived from our evaluation (60.54). However, the results of CSFLOW, DREAM, DSR, FASTFLOW, FRE, PaDiM, STFPM, and UFLOW have relatively small differences from our assessment. In contrast, CFLOW, DFM, PatchCore, and RD are in perfect agreement with our results.

9. Relevant links

IADBE Core
IADBE Server
IADBE Frontend
IADBE Pre-trained Models
IADBE Custom Dataset
Inference with Gradio deployed on HF spaces
Final Version Paper

10. Future work

Extending User Interaction with Gradio
Extended integration with Huggingface
Simplify front- and back-office with Streamlit
Beautify benchmarking visualizations
Implementation of the IADBE system pipeline
Multi-class harmonized model benchmarking
Reproducing high-performance models based on Anomalib
Enhanced support for video datasets based on Anomalib

11. Miscellaneous

I'm about to finish my master's thesis in computer science at the Technical University of Darmstadt and it's really hard! Students and industrialists are welcome to use and build on my cross-platform/language project for secondary development, and I hope it will provide you with some inspiration.

Meanwhile, you are warmly welcome to join the IADBE open source project and work together to promote the expansion of the industrial anomaly detection benchmark engine. You can submit Issues if you encounter problems, and if you want to contribute, you are welcome to participate in the development via Pull Requests!