用大模型提升图像分布外检测,兼顾远距离与近距离异常
Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
- 用大语言模型生成类别背景描述,结合CLIP提取特征
- 通过减去背景特征,精准提取正常数据核心语义
- 在多个数据集上性能超越现有方法,抗域偏移能力强
分布外(OOD)检测近年来借助强大的视觉-语言模型(如CLIP)实现了零样本进展。然而,现有研究多聚焦于远距离OOD性能提升,可能牺牲近距OOD效果。为此,本文提出一种新策略,通过创新性融合大语言模型(LLM)与视觉-语言模型(VLM),同时提升远、近距离OOD检测性能。首先利用LLM生成正常类别的超类及其背景描述,再通过CLIP提取特征;随后从超类特征中减去背景特征,获得正常数据的核心语义表示。该表示用于从WordNet候选标签集中筛选更合适的负样本,从而提升零样本OOD检测性能。此外,引入新型少样本提示调优与视觉提示调优,使框架更好地适配目标分布。实验表明,该方法在多个基准上持续优于当前最优方法,最高提升AUROC达2.9%,降低FPR95达12.6%。同时,在不同领域间表现出更强的抗协变量偏移能力,凸显其在真实场景中的有效性。
原文摘要 · Abstract (English)
Out-of-distribution (OOD) detection has seen significant advancements with zero-shot approaches by leveraging the powerful Vision-Language Models (VLMs) such as CLIP. However, prior research works have predominantly focused on enhancing Far-OOD performance, while potentially compromising Near-OOD efficacy, as observed from our pilot study. To address this issue, we propose a novel strategy to enhance zero-shot OOD detection performances for both Far-OOD and Near-OOD scenarios by innovatively harnessing Large Language Models (LLMs) and VLMs. Our approach first exploit an LLM to generate superclasses of the ID labels and their corresponding background descriptions followed by feature extraction using CLIP. We then isolate the core semantic features for ID data by subtracting background features from the superclass features. The refined representation facilitates the selection of more appropriate negative labels for OOD data from a comprehensive candidate label set of WordNet, thereby enhancing the performance of zero-shot OOD detection in both scenarios. Furthermore, we introduce novel few-shot prompt tuning and visual prompt tuning to adapt the proposed framework to better align with the target distribution. Experimental results demonstrate that the proposed approach consistently outperforms current state-of-the-art methods across multiple benchmarks, with an improvement of up to 2.9% in AUROC and a reduction of up to 12.6% in FPR95. Additionally, our method exhibits superior robustness against covariate shift across different domains, further highlighting its effectiveness in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。