用DINOv3做零样本异常检测,通过校准注意力提升识别精度。
AD-DINOv3: Enhancing DINOv3 for Zero-Shot Anomaly Detection with Anomaly-Aware Calibration
- 将DINOv3与文本编码器结合,构建多模态对比学习框架。
- 在8个工业和医疗数据集上优于现有方法,最高提升2.1%。
- 新增异常感知校准模块,让模型关注异常区域而非正常物体。
零样本异常检测(ZSAD)旨在识别任意新类别中的异常,具有可扩展性和低标注成本优势。传统方法多基于CLIP模型,通过视觉与文本嵌入的相似性进行检测。近年来,DINOv3等视觉基础模型展现出强大的可迁移表征能力。本文首次将DINOv3应用于ZSAD,但面临两大挑战:(i) 大规模预训练数据与异常检测任务间的领域偏差导致特征错位;(ii) 预训练表示对全局语义的固有偏好,常使细微异常被误判为正常前景部分。为此,我们提出AD-DINOv3,一种新型多模态框架。具体地,将异常检测建模为多模态对比学习问题,以DINOv3为视觉主干提取块级令牌与CLS令牌,使用CLIP文本编码器生成正常与异常提示的嵌入。为弥合领域差距,引入轻量级适配器,实现双模态表征的异常检测任务重校准。在此基础上,设计异常感知校准模块(AACM),显式引导CLS令牌关注异常区域而非通用前景语义,从而增强判别力。在八个工业与医疗基准上的大量实验表明,AD-DINOv3持续达到或超越当前最优方法。代码将开源于https://github.com/Kaisor-Yuan/AD-DINOv3。
原文摘要 · Abstract (English)
Zero-Shot Anomaly Detection (ZSAD) seeks to identify anomalies from arbitrary novel categories, offering a scalable and annotation-efficient solution. Traditionally, most ZSAD works have been based on the CLIP model, which performs anomaly detection by calculating the similarity between visual and text embeddings. Recently, vision foundation models such as DINOv3 have demonstrated strong transferable representation capabilities. In this work, we are the first to adapt DINOv3 for ZSAD. However, this adaptation presents two key challenges: (i) the domain bias between large-scale pretraining data and anomaly detection tasks leads to feature misalignment; and (ii) the inherent bias toward global semantics in pretrained representations often leads to subtle anomalies being misinterpreted as part of the normal foreground objects, rather than being distinguished as abnormal regions. To overcome these challenges, we introduce AD-DINOv3, a novel vision-language multimodal framework designed for ZSAD. Specifically, we formulate anomaly detection as a multimodal contrastive learning problem, where DINOv3 is employed as the visual backbone to extract patch tokens and a CLS token, and the CLIP text encoder provides embeddings for both normal and abnormal prompts. To bridge the domain gap, lightweight adapters are introduced in both modalities, enabling their representations to be recalibrated for the anomaly detection task. Beyond this baseline alignment, we further design an Anomaly-Aware Calibration Module (AACM), which explicitly guides the CLS token to attend to anomalous regions rather than generic foreground semantics, thereby enhancing discriminability. Extensive experiments on eight industrial and medical benchmarks demonstrate that AD-DINOv3 consistently matches or surpasses state-of-the-art methods.The code will be available at https://github.com/Kaisor-Yuan/AD-DINOv3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。