SlideShare a Scribd company logo
2020/07/19
Ho Seong Lee (hoya012)
Cognex Deep Learning Lab
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 1
Contents
• Introduction
• Related Work
• Datasets for Anomaly Detection in natural images
• Methods
• MVTec-AD Dataset
• Benchmark
• Discussion
• Conclusion
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 2
Introduction
What is Anomaly Detection?
• Detecting anomalous regions in images or videos or time-series data
• Today, we will focus anomaly detection in industrial images
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 3
Reference: https://hoya012.github.io/blog/anomaly-detection-overview-1/
Introduction
Definition of Anomaly Detection
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 4
• I wrote technical blog post about anomaly detection (Korean Only TT)
• I recommend reading this post if you want to know more about anomaly detection
Reference: https://hoya012.github.io/blog/anomaly-detection-overview-1/
Introduction
Anomaly Detection in PR-12
• Unsupervised Anomaly Detection 1, One-Class Anomaly Detection 1, Out-of-distribution Detection 2
• Minor topic.. 5/263 = 1.9%
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 5
PR-115
By 강민국님
PR-148
By 강민국님
PR-190
By 강민국님
PR-235
By 이도엽님
Related Work
Datasets for Anomaly Detection in natural images
• One-Class Anomaly Detection: MNIST, CIFAR-10
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 6
Normal Class Anomaly Class
Provide a large amount of
train/test data
Anomaly samples differ
significantly from normal
samples
Related Work
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 7
Datasets for Anomaly Detection in natural images
• Unsupervised Anomaly Detection (Segmentation): DAGM 2007, NanoTWICE
Only focus on the
inspection of textured
surfaces
Only 5 defect-free images
can be used for training
Defects were generated
by similar texture models
Related Work
Methods - Generative Adversarial Networks (AnoGAN) → See PR-115!
• Train GAN using normal samples, and fix Generator and Discriminator
• Search for a latent sample that reproduces a given input image and manages to fool discriminator
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 8
Reference: “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery”
Related Work
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 9
Methods – Deep Convolutional Autoencoders and Variational Autoencoders (VAE)
• Reconstruct normal training samples through a bottleneck (latent space)
• During testing, they fail to reproduce images that differ from the data that was observed during training
Many papers provide further
evidence that probabilities obtained
from VAEs and other deep generative
models might fail to model the true
likelihood of the training data
Reference: https://hoya012.github.io/blog/anomaly-detection-overview-1/
Related Work
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 10
Methods – Features of pre-trained CNN (Feature Dictionary)
• Use feature descriptors obtained from ImageNet pre-trained CNN
• Sliding window-based approach → for large images, very slow
Reference: “Anomaly Detection in Nanofibrous Materials by CNN-Based Self Similarity”
Related Work
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 11
Methods – Traditional Methods
• Use hand-crafted feature descriptors from defect-free texture images
• Gaussian Mixture Model(GMM)-based Texture Inspection Model for texture objects
• Shape-based Matching-based Variation Model for non-texture objects
MVTec-AD Dataset
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
MVTec-AD Dataset Description
• 15 categories with 3629 images for training and validation and 1725 images for testing
• The defects were generated with the aim to produce realistic anomalies as they would occur in real-
world industrial inspection scenarios
12
OK OK / NG
MVTec-AD Dataset
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 13
Grid, Screw, Zipper = Gray(1ch)
MVTec-AD Dataset
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 14
MVTec-AD Dataset Description
• All images were acquired using a 2048 x 2048 pixel high-resolution industrial RGB sensor
• Afterwards, the images were cropped to a suitable output size (700 x 700 ~ 1024 x 1024)
• The images were acquired under highly controlled illumination conditions. For some object classes,
however, the illumination was altered intentionally to increase variability
• Provide pixel-precise ground truth labels for each defective image region
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 15
Evaluated Methods
• AnoGAN: use the publicly available implementation from DoYup Lee!! (PR-12 Presenter)
• L2 and SSIM Autoencoder: same CAE architecture as MVTec’s previous paper
• CNN Feature Dictionary: 512-dimensional avgpool layer of ResNet-18 pretrained on ImageNet
• GMM-Based Texture Inspection Model / Variation Model: use HALCON machine vision library from
MVTec
MVTec CAE
→ www.github.com/LeeDoYup/AnoGAN-tf
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 16
Data Augmentation
• Texture images: random crop rotated rectangular patches of fixed size
• Object images: random translation and rotation and additional random flip if possible
Texture Object
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 17
Evaluation
• Output score is one-channel spatial map. To obtain a final segmentation result, a threshold must be
determined! Use defect-free validation images to estimate the threshold
Ideal threshold.. But.. we don’t know
Original Reconstructed Anomaly Map
Reference: SUALAB Research Team Park’s Figure
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 18
Evaluation
• Output score is one-channel spatial map. To obtain a final segmentation result, a threshold must be
determined! Use defect-free validation images to estimate the threshold
• For every category, define a minimum defect area that a connected component in the thresholded
anomaly map must have to be classified as a defective region
• Successively segment the anomaly maps of the defect-free validation images with increasing thresholds
• The threshold that yielded this segmentation is used for further evaluation
Human’s prior
Maybe.. Blob?
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 19
Evaluation Metric
• For classification, use the accuracy of correctly classified test images
• For segmentation, use relative per-region overlap (IoU) and AUROC (which is independent of threshold)
Red line: GT / Green map: Prediction
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 20
Results
Classification Segmentation
Top row: OK
Bottom row: NG
Top row: IoU
Bottom row: AUROC
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 21
Results
• For texture images, none of methods emerges as a clear winner
• Autoencoder and CNN Feature Dictionary perform well
Benchmark
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 22
Results
• For object images, Autoencoder achieve the best results
• L2 AE achieves better per-region overlap values
Discussion
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
AnoGAN
• Observe a tendency of GAN training to result in mode collapse
• AnoGAN has great difficulties with object categories for which the objects appear in various
shapes or orientations in the dataset
23
Discussion
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
Autoencoder
• Observe stable training across all dataset categories both SSIM AE and L2 AE
• For some categories, however, fail to model small details, which results in rather blurry reconstruction
• This is especially the case for high-frequency textures, which appear in tile and zipper
24
False Positive!
Discussion
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
CNN Feature Dictionary
• As a method proposed for detection of anomalous regions in textured surfaces, CNN Feature
Dictionary achieves satisfactory results for all textures except grid
• Its performance degenerates when evaluated on objects categories
25
False Negative!
Discussion
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
Others (Texture Inspection Model and Variation Model)
• Good performance and Bad performance..
26
Texture
Inspection
Variation
Model
Conclusion
PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
• Introduce MVTec Anomaly Detection dataset (MVTec-AD), a novel dataset for unsupervised
anomaly detection mimicking real-world industrial inspection scenarios
• Several methods were thoroughly evaluated on this dataset
• The evaluations provide a first benchmark on this dataset and show that there is still
considerable room for improvement
→ The first real-world industrial unsupervised anomaly detection dataset!
→ Some ambiguous evaluation metric(threshold).. And didn’t provide source code
27

More Related Content

What's hot (20)

PPTX
Artificial neural network
Mohd Arafat Shaikh
 
PPTX
Biological Neural Network.pptx
Abdul Rehman
 
PPTX
[BMVC 2022] DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detec...
Ziyuan Zhao
 
PPTX
Recurrent neural network
Syed Annus Ali SHah
 
PDF
capsule network
민기 정
 
PPTX
Convolutional Neural Networks
Ashray Bhandare
 
PPTX
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Simplilearn
 
PPTX
CNN Tutorial
Sungjoon Choi
 
PDF
Single Image Super Resolution Overview
LEE HOSEONG
 
PPTX
Long Short Term Memory LSTM
Abdullah al Mamun
 
PDF
Recurrent Neural Networks
Rakuten Group, Inc.
 
PPTX
Depthwise separable convolution
Dong-Won Shin
 
PDF
Recurrent neural networks
Viacheslav Khomenko
 
PPTX
Inductive analytical approaches to learning
swapnac12
 
PPTX
ConvNeXt: A ConvNet for the 2020s explained
Sushant Gautam
 
PPTX
Gradient descent method
Prof. Neeta Awasthy
 
PPTX
Ip packet delivery
rajisri2
 
PPTX
Artificial Neural Network Topology
Harshana Madusanka Jayamaha
 
PDF
Object Detetcion using SSD-MobileNet
IRJET Journal
 
PDF
Convolutional neural network
Yan Xu
 
Artificial neural network
Mohd Arafat Shaikh
 
Biological Neural Network.pptx
Abdul Rehman
 
[BMVC 2022] DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detec...
Ziyuan Zhao
 
Recurrent neural network
Syed Annus Ali SHah
 
capsule network
민기 정
 
Convolutional Neural Networks
Ashray Bhandare
 
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Simplilearn
 
CNN Tutorial
Sungjoon Choi
 
Single Image Super Resolution Overview
LEE HOSEONG
 
Long Short Term Memory LSTM
Abdullah al Mamun
 
Recurrent Neural Networks
Rakuten Group, Inc.
 
Depthwise separable convolution
Dong-Won Shin
 
Recurrent neural networks
Viacheslav Khomenko
 
Inductive analytical approaches to learning
swapnac12
 
ConvNeXt: A ConvNet for the 2020s explained
Sushant Gautam
 
Gradient descent method
Prof. Neeta Awasthy
 
Ip packet delivery
rajisri2
 
Artificial Neural Network Topology
Harshana Madusanka Jayamaha
 
Object Detetcion using SSD-MobileNet
IRJET Journal
 
Convolutional neural network
Yan Xu
 

Similar to MVTec AD: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection (20)

PDF
PhdThesis_presentation
Xavier Gibert
 
PDF
AIFrienz_Webinar_Tomomi_Research_Inc).pdf
Seong-Hun Choe
 
PPTX
[NS][Lab_Seminar_250203]KAG-prompt (1).pptx
thanhdowork
 
PDF
Anomaly detection using deep learning based model with feature attention
IAESIJAI
 
PDF
Real-time anomaly detection in electric motor operation noise
IAESIJAI
 
PDF
Anomaly Detection using Deep Auto-Encoders
Gianmario Spacagna
 
PDF
Enhancing Anomaly Detection Accuracy with Conditional CutPaste
MILab
 
PDF
“Visual Anomaly Detection with FOMO-AD,” a Presentation from Edge Impulse
Edge AI and Vision Alliance
 
PDF
Optimizing CNN-BiGRU Performance: Mish Activation and Comparative Analysis
IJCNCJournal
 
PDF
Optimizing CNN-BiGRU Performance: Mish Activation and Comparative Analysis
IJCNCJournal
 
PDF
Optimizing CNN-BiGRU Performance: Mish Activation and Comparative Analysis
IJCNCJournal
 
PPTX
Anomaly Detection with Azure and .net
Marco Parenzan
 
PDF
Detecting anomalies in security cameras with 3D-convolutional neural network ...
IJECEIAES
 
PDF
2007.02500.pdf
TadiyosHailemichael
 
PDF
Partial Object Detection in Inclined Weather Conditions
IRJET Journal
 
PDF
An Overview of Various Techniques Involved in Detection of Anomalies from Sur...
ijcseit
 
PDF
INTRODUCTION OF A NOVEL ANOMALOUS SOUND DETECTION METHODOLOGY
ijsc
 
PDF
Introduction of a Novel Anomalous Sound Detection Methodology
ijsc
 
PDF
Deep learning and image analytics using Python by Dr Sanparit
BAINIDA
 
PPTX
Acoustic Anomaly Detection Using Deep Autoencoders
Jan-Philipp Simen
 
PhdThesis_presentation
Xavier Gibert
 
AIFrienz_Webinar_Tomomi_Research_Inc).pdf
Seong-Hun Choe
 
[NS][Lab_Seminar_250203]KAG-prompt (1).pptx
thanhdowork
 
Anomaly detection using deep learning based model with feature attention
IAESIJAI
 
Real-time anomaly detection in electric motor operation noise
IAESIJAI
 
Anomaly Detection using Deep Auto-Encoders
Gianmario Spacagna
 
Enhancing Anomaly Detection Accuracy with Conditional CutPaste
MILab
 
“Visual Anomaly Detection with FOMO-AD,” a Presentation from Edge Impulse
Edge AI and Vision Alliance
 
Optimizing CNN-BiGRU Performance: Mish Activation and Comparative Analysis
IJCNCJournal
 
Optimizing CNN-BiGRU Performance: Mish Activation and Comparative Analysis
IJCNCJournal
 
Optimizing CNN-BiGRU Performance: Mish Activation and Comparative Analysis
IJCNCJournal
 
Anomaly Detection with Azure and .net
Marco Parenzan
 
Detecting anomalies in security cameras with 3D-convolutional neural network ...
IJECEIAES
 
2007.02500.pdf
TadiyosHailemichael
 
Partial Object Detection in Inclined Weather Conditions
IRJET Journal
 
An Overview of Various Techniques Involved in Detection of Anomalies from Sur...
ijcseit
 
INTRODUCTION OF A NOVEL ANOMALOUS SOUND DETECTION METHODOLOGY
ijsc
 
Introduction of a Novel Anomalous Sound Detection Methodology
ijsc
 
Deep learning and image analytics using Python by Dr Sanparit
BAINIDA
 
Acoustic Anomaly Detection Using Deep Autoencoders
Jan-Philipp Simen
 
Ad

More from LEE HOSEONG (20)

PDF
Unsupervised anomaly detection using style distillation
LEE HOSEONG
 
PDF
do adversarially robust image net models transfer better
LEE HOSEONG
 
PDF
CNN Architecture A to Z
LEE HOSEONG
 
PDF
carrier of_tricks_for_image_classification
LEE HOSEONG
 
PDF
"The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Gen...
LEE HOSEONG
 
PDF
Mixed Precision Training Review
LEE HOSEONG
 
PDF
YOLOv4: optimal speed and accuracy of object detection review
LEE HOSEONG
 
PDF
FixMatch:simplifying semi supervised learning with consistency and confidence
LEE HOSEONG
 
PDF
"Revisiting self supervised visual representation learning" Paper Review
LEE HOSEONG
 
PDF
Unsupervised visual representation learning overview: Toward Self-Supervision
LEE HOSEONG
 
PDF
Human uncertainty makes classification more robust, ICCV 2019 Review
LEE HOSEONG
 
PDF
2019 ICLR Best Paper Review
LEE HOSEONG
 
PDF
2019 cvpr paper_overview
LEE HOSEONG
 
PDF
"Google Vizier: A Service for Black-Box Optimization" Paper Review
LEE HOSEONG
 
PDF
"Searching for Activation Functions" Paper Review
LEE HOSEONG
 
PDF
"Learning transferable architectures for scalable image recognition" Paper Re...
LEE HOSEONG
 
PDF
"Learning From Noisy Large-Scale Datasets With Minimal Supervision" Paper Review
LEE HOSEONG
 
PDF
"Dataset and metrics for predicting local visible differences" Paper Review
LEE HOSEONG
 
PDF
"From image level to pixel-level labeling with convolutional networks" Paper ...
LEE HOSEONG
 
PDF
"simple does it weakly supervised instance and semantic segmentation" Paper r...
LEE HOSEONG
 
Unsupervised anomaly detection using style distillation
LEE HOSEONG
 
do adversarially robust image net models transfer better
LEE HOSEONG
 
CNN Architecture A to Z
LEE HOSEONG
 
carrier of_tricks_for_image_classification
LEE HOSEONG
 
"The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Gen...
LEE HOSEONG
 
Mixed Precision Training Review
LEE HOSEONG
 
YOLOv4: optimal speed and accuracy of object detection review
LEE HOSEONG
 
FixMatch:simplifying semi supervised learning with consistency and confidence
LEE HOSEONG
 
"Revisiting self supervised visual representation learning" Paper Review
LEE HOSEONG
 
Unsupervised visual representation learning overview: Toward Self-Supervision
LEE HOSEONG
 
Human uncertainty makes classification more robust, ICCV 2019 Review
LEE HOSEONG
 
2019 ICLR Best Paper Review
LEE HOSEONG
 
2019 cvpr paper_overview
LEE HOSEONG
 
"Google Vizier: A Service for Black-Box Optimization" Paper Review
LEE HOSEONG
 
"Searching for Activation Functions" Paper Review
LEE HOSEONG
 
"Learning transferable architectures for scalable image recognition" Paper Re...
LEE HOSEONG
 
"Learning From Noisy Large-Scale Datasets With Minimal Supervision" Paper Review
LEE HOSEONG
 
"Dataset and metrics for predicting local visible differences" Paper Review
LEE HOSEONG
 
"From image level to pixel-level labeling with convolutional networks" Paper ...
LEE HOSEONG
 
"simple does it weakly supervised instance and semantic segmentation" Paper r...
LEE HOSEONG
 
Ad

Recently uploaded (20)

PPTX
AI Code Generation Risks (Ramkumar Dilli, CIO, Myridius)
Priyanka Aash
 
PPTX
Farrell_Programming Logic and Design slides_10e_ch02_PowerPoint.pptx
bashnahara11
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PDF
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
PDF
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
NewMind AI Weekly Chronicles – July’25, Week III
NewMind AI
 
PDF
RAT Builders - How to Catch Them All [DeepSec 2024]
malmoeb
 
PDF
Researching The Best Chat SDK Providers in 2025
Ray Fields
 
PPTX
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
PDF
Market Insight : ETH Dominance Returns
CIFDAQ
 
PPTX
The Future of AI & Machine Learning.pptx
pritsen4700
 
PDF
introduction to computer hardware and sofeware
chauhanshraddha2007
 
PDF
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
PDF
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
PPTX
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
PDF
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
PPTX
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PPTX
AVL ( audio, visuals or led ), technology.
Rajeshwri Panchal
 
AI Code Generation Risks (Ramkumar Dilli, CIO, Myridius)
Priyanka Aash
 
Farrell_Programming Logic and Design slides_10e_ch02_PowerPoint.pptx
bashnahara11
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
NewMind AI Weekly Chronicles – July’25, Week III
NewMind AI
 
RAT Builders - How to Catch Them All [DeepSec 2024]
malmoeb
 
Researching The Best Chat SDK Providers in 2025
Ray Fields
 
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
Market Insight : ETH Dominance Returns
CIFDAQ
 
The Future of AI & Machine Learning.pptx
pritsen4700
 
introduction to computer hardware and sofeware
chauhanshraddha2007
 
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
AVL ( audio, visuals or led ), technology.
Rajeshwri Panchal
 

MVTec AD: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection

  • 1. 2020/07/19 Ho Seong Lee (hoya012) Cognex Deep Learning Lab PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 1
  • 2. Contents • Introduction • Related Work • Datasets for Anomaly Detection in natural images • Methods • MVTec-AD Dataset • Benchmark • Discussion • Conclusion PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 2
  • 3. Introduction What is Anomaly Detection? • Detecting anomalous regions in images or videos or time-series data • Today, we will focus anomaly detection in industrial images PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 3 Reference: https://hoya012.github.io/blog/anomaly-detection-overview-1/
  • 4. Introduction Definition of Anomaly Detection PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 4 • I wrote technical blog post about anomaly detection (Korean Only TT) • I recommend reading this post if you want to know more about anomaly detection Reference: https://hoya012.github.io/blog/anomaly-detection-overview-1/
  • 5. Introduction Anomaly Detection in PR-12 • Unsupervised Anomaly Detection 1, One-Class Anomaly Detection 1, Out-of-distribution Detection 2 • Minor topic.. 5/263 = 1.9% PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 5 PR-115 By 강민국님 PR-148 By 강민국님 PR-190 By 강민국님 PR-235 By 이도엽님
  • 6. Related Work Datasets for Anomaly Detection in natural images • One-Class Anomaly Detection: MNIST, CIFAR-10 PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 6 Normal Class Anomaly Class Provide a large amount of train/test data Anomaly samples differ significantly from normal samples
  • 7. Related Work PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 7 Datasets for Anomaly Detection in natural images • Unsupervised Anomaly Detection (Segmentation): DAGM 2007, NanoTWICE Only focus on the inspection of textured surfaces Only 5 defect-free images can be used for training Defects were generated by similar texture models
  • 8. Related Work Methods - Generative Adversarial Networks (AnoGAN) → See PR-115! • Train GAN using normal samples, and fix Generator and Discriminator • Search for a latent sample that reproduces a given input image and manages to fool discriminator PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 8 Reference: “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery”
  • 9. Related Work PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 9 Methods – Deep Convolutional Autoencoders and Variational Autoencoders (VAE) • Reconstruct normal training samples through a bottleneck (latent space) • During testing, they fail to reproduce images that differ from the data that was observed during training Many papers provide further evidence that probabilities obtained from VAEs and other deep generative models might fail to model the true likelihood of the training data Reference: https://hoya012.github.io/blog/anomaly-detection-overview-1/
  • 10. Related Work PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 10 Methods – Features of pre-trained CNN (Feature Dictionary) • Use feature descriptors obtained from ImageNet pre-trained CNN • Sliding window-based approach → for large images, very slow Reference: “Anomaly Detection in Nanofibrous Materials by CNN-Based Self Similarity”
  • 11. Related Work PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 11 Methods – Traditional Methods • Use hand-crafted feature descriptors from defect-free texture images • Gaussian Mixture Model(GMM)-based Texture Inspection Model for texture objects • Shape-based Matching-based Variation Model for non-texture objects
  • 12. MVTec-AD Dataset PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection MVTec-AD Dataset Description • 15 categories with 3629 images for training and validation and 1725 images for testing • The defects were generated with the aim to produce realistic anomalies as they would occur in real- world industrial inspection scenarios 12 OK OK / NG
  • 13. MVTec-AD Dataset PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 13 Grid, Screw, Zipper = Gray(1ch)
  • 14. MVTec-AD Dataset PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 14 MVTec-AD Dataset Description • All images were acquired using a 2048 x 2048 pixel high-resolution industrial RGB sensor • Afterwards, the images were cropped to a suitable output size (700 x 700 ~ 1024 x 1024) • The images were acquired under highly controlled illumination conditions. For some object classes, however, the illumination was altered intentionally to increase variability • Provide pixel-precise ground truth labels for each defective image region
  • 15. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 15 Evaluated Methods • AnoGAN: use the publicly available implementation from DoYup Lee!! (PR-12 Presenter) • L2 and SSIM Autoencoder: same CAE architecture as MVTec’s previous paper • CNN Feature Dictionary: 512-dimensional avgpool layer of ResNet-18 pretrained on ImageNet • GMM-Based Texture Inspection Model / Variation Model: use HALCON machine vision library from MVTec MVTec CAE → www.github.com/LeeDoYup/AnoGAN-tf
  • 16. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 16 Data Augmentation • Texture images: random crop rotated rectangular patches of fixed size • Object images: random translation and rotation and additional random flip if possible Texture Object
  • 17. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 17 Evaluation • Output score is one-channel spatial map. To obtain a final segmentation result, a threshold must be determined! Use defect-free validation images to estimate the threshold Ideal threshold.. But.. we don’t know Original Reconstructed Anomaly Map Reference: SUALAB Research Team Park’s Figure
  • 18. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 18 Evaluation • Output score is one-channel spatial map. To obtain a final segmentation result, a threshold must be determined! Use defect-free validation images to estimate the threshold • For every category, define a minimum defect area that a connected component in the thresholded anomaly map must have to be classified as a defective region • Successively segment the anomaly maps of the defect-free validation images with increasing thresholds • The threshold that yielded this segmentation is used for further evaluation Human’s prior Maybe.. Blob?
  • 19. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 19 Evaluation Metric • For classification, use the accuracy of correctly classified test images • For segmentation, use relative per-region overlap (IoU) and AUROC (which is independent of threshold) Red line: GT / Green map: Prediction
  • 20. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 20 Results Classification Segmentation Top row: OK Bottom row: NG Top row: IoU Bottom row: AUROC
  • 21. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 21 Results • For texture images, none of methods emerges as a clear winner • Autoencoder and CNN Feature Dictionary perform well
  • 22. Benchmark PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection 22 Results • For object images, Autoencoder achieve the best results • L2 AE achieves better per-region overlap values
  • 23. Discussion PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection AnoGAN • Observe a tendency of GAN training to result in mode collapse • AnoGAN has great difficulties with object categories for which the objects appear in various shapes or orientations in the dataset 23
  • 24. Discussion PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection Autoencoder • Observe stable training across all dataset categories both SSIM AE and L2 AE • For some categories, however, fail to model small details, which results in rather blurry reconstruction • This is especially the case for high-frequency textures, which appear in tile and zipper 24 False Positive!
  • 25. Discussion PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection CNN Feature Dictionary • As a method proposed for detection of anomalous regions in textured surfaces, CNN Feature Dictionary achieves satisfactory results for all textures except grid • Its performance degenerates when evaluated on objects categories 25 False Negative!
  • 26. Discussion PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection Others (Texture Inspection Model and Variation Model) • Good performance and Bad performance.. 26 Texture Inspection Variation Model
  • 27. Conclusion PR-263 | MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection • Introduce MVTec Anomaly Detection dataset (MVTec-AD), a novel dataset for unsupervised anomaly detection mimicking real-world industrial inspection scenarios • Several methods were thoroughly evaluated on this dataset • The evaluations provide a first benchmark on this dataset and show that there is still considerable room for improvement → The first real-world industrial unsupervised anomaly detection dataset! → Some ambiguous evaluation metric(threshold).. And didn’t provide source code 27