用CLIP和扩散模型融合检测异常,兼顾全局与局部特征。
CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection
- 结合CLIP的全局识别与扩散模型的局部建模,形成互补。
- 在MVTec-AD和VisA数据集上分割与分类均优于基线方法。
- 适合工业缺陷检测等需要高精度异常识别的场景。
异常检测因异常定义模糊、类型多样(如局部与全局缺陷)以及训练数据稀缺而复杂。为此,我们提出CLIPFUSION方法,融合判别式与生成式基础模型:基于CLIP的判别模型擅长捕捉全局特征,扩散模型则有效建模局部细节,实现协同互补。特别地,我们提出利用扩散模型中的交叉注意力图与特征图进行异常检测的新方法。在MVTec-AD和VisA等基准数据集上的实验表明,CLIPFUSION在异常分割与分类任务中持续优于基线方法,性能优异。该工作验证了多模态、多模型融合在应对异常检测多重挑战中的有效性,为实际应用提供了可扩展的解决方案。
原文摘要 · Abstract (English)
Anomaly detection is a complex problem due to the ambiguity in defining anomalies, the diversity of anomaly types (e.g., local and global defect), and the scarcity of training data. As such, it necessitates a comprehensive model capable of capturing both low-level and high-level features, even with limited data. To address this, we propose CLIPFUSION, a method that leverages both discriminative and generative foundation models. Specifically, the CLIP-based discriminative model excels at capturing global features, while the diffusion-based generative model effectively captures local details, creating a synergistic and complementary approach. Notably, we introduce a methodology for utilizing cross-attention maps and feature maps extracted from diffusion models specifically for anomaly detection. Experimental results on benchmark datasets (MVTec-AD, VisA) demonstrate that CLIPFUSION consistently outperforms baseline methods, achieving outstanding performance in both anomaly segmentation and classification. We believe that our method underscores the effectiveness of multi-modal and multi-model fusion in tackling the multifaceted challenges of anomaly detection, providing a scalable solution for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。