用视觉语言模型增强扩散模型，实现无需标注的多类异常检测。
VLMDiff: Leveraging Vision-Language Models for Multi-Class Anomaly Detection with Diffusion
- 用预训练VLM生成图像描述，作为扩散模型的条件输入。
- 在Real-IAD上像素级PRO提升25点，在COCO-AD上提升8点。
- 无需每类单独训练，适合真实世界多类别异常场景。
在多样化的多类别真实图像中检测视觉异常是一项重大挑战。本文提出一种新的无监督多类别视觉异常检测框架，将潜在扩散模型（LDM）与视觉语言模型（VLM）结合，提升异常定位与检测能力。具体地，利用预训练VLM通过简单提示获取图像详细描述，作为LDM训练的额外条件。现有基于扩散的方法依赖合成噪声生成，限制了泛化能力且需每类单独训练，难以扩展。本文方法通过VLM获取正常图像描述，无需人工标注或额外训练，这些描述作为条件输入扩散模型，学习鲁棒的正常图像特征表示，实现多类别异常检测。实验表明，该方法在Real-IAD数据集上像素级区域重叠（PRO）指标最高提升25点，在COCO-AD数据集上提升8点，优于现有最优扩散方法。代码已公开于https://github.com/giddyyupp/VLMDiff。
原文摘要 · Abstract (English)
Detecting visual anomalies in diverse, multi-class real-world images is a significant challenge. We introduce \ours, a novel unsupervised multi-class visual anomaly detection framework. It integrates a Latent Diffusion Model (LDM) with a Vision-Language Model (VLM) for enhanced anomaly localization and detection. Specifically, a pre-trained VLM with a simple prompt extracts detailed image descriptions, serving as additional conditioning for LDM training. Current diffusion-based methods rely on synthetic noise generation, limiting their generalization and requiring per-class model training, which hinders scalability. \ours, however, leverages VLMs to obtain normal captions without manual annotations or additional training. These descriptions condition the diffusion model, learning a robust normal image feature representation for multi-class anomaly detection. Our method achieves competitive performance, improving the pixel-level Per-Region-Overlap (PRO) metric by up to 25 points on the Real-IAD dataset and 8 points on the COCO-AD dataset, outperforming state-of-the-art diffusion-based approaches. Code is available at https://github.com/giddyyupp/VLMDiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。