用CLIP+异常描述符，实现少样本工业缺陷分类
CLIP-FSAC++: Few-Shot Anomaly Classification with Anomaly Descriptor Based on CLIP
- 引入跨模态交互模块，生成图像和文本的专属嵌入
- 在1-8张正常样本下，VisA和MVTec-AD上性能领先
- 适合缺乏标注数据的工业缺陷检测场景
工业异常分类是制造过程中保障产品质量与安全的关键任务。针对工业场景中数据稀缺的问题，近期涌现出大量少样本异常检测方法。本文提出一种单阶段训练的少样本异常分类框架——CLIP-FSAC++。具体而言，我们在图像和文本编码器后引入名为异常描述符（Anomaly Descriptor）的跨模态交互模块，增强视觉与文本嵌入的相关性，并将CLIP的预训练表示适配至目标数据分布。该模块包含图像到文本的交叉注意力机制，用于生成图像特定的文本嵌入；以及文本到图像的交叉注意力机制，用于生成文本特定的视觉嵌入。这些模态特定的嵌入用于增强原始CLIP表示，提升匹配能力。我们在VisA和MVTec-AD数据集上，对1、2、4、8张正常样本下的少样本异常分类任务进行了全面实验评估。源代码已开源。
原文摘要 · Abstract (English)
Industrial anomaly classification (AC) is an indispensable task in industrial manufacturing, which guarantees quality and safety of various product. To address the scarcity of data in industrial scenarios, lots of few-shot anomaly detection methods emerge recently. In this paper, we propose an effective few-shot anomaly classification (FSAC) framework with one-stage training, dubbed CLIP-FSAC++. Specifically, we introduce a cross-modality interaction module named Anomaly Descriptor following image and text encoders, which enhances the correlation of visual and text embeddings and adapts the representations of CLIP from pre-trained data to target data. In anomaly descriptor, image-to-text cross-attention module is used to obtain image-specific text embeddings and text-to-image cross-attention module is used to obtain text-specific visual embeddings. Then these modality-specific embeddings are used to enhance original representations of CLIP for better matching ability. Comprehensive experiment results are provided for evaluating our method in few-normal shot anomaly classification on VisA and MVTEC-AD for 1, 2, 4 and 8-shot settings. The source codes are at https://github.com/Jay-zzcoder/clip-fsac-pp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。