arXiv:2409.04796cs.CV2024-09ICLR被引 21

通过局部提示增强,提升视觉语言模型少样本异常检测能力。

Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection

  • 冻结全局提示,用可训练局部提示进行细粒度区域优化。
  • 4次示例下,FPR95降低5.17%,优于以往16次示例结果。
  • 适用于需要少样本高效检测异常的视觉任务场景。

异常分布(OOD)检测旨在区分未知类别与已知类别,在实际应用中备受关注。近期视觉语言模型(VLM)的发展推动了基于少样本微调的VLM OOD检测研究。然而,现有方法多聚焦于优化全局提示,忽略了对异常局部信息的精细利用。为此,我们冻结全局提示,提出一种新的粗到精微调范式——Local-Prompt,强调局部提示对区域特征的增强。该方法包含两个核心组件:全局提示引导的负向增强,以及局部提示驱动的区域正则化。前者利用冻结的粗粒度全局提示作为引导,引入负向增强以挖掘局部异常知识;后者通过可训练的局部提示和区域正则化有效捕捉局部特征,辅助异常识别。同时,我们设计了区域相关度量,进一步丰富检测能力。由于仅优化局部提示,该方法可无缝集成已有训练好的全局提示,在推理阶段提升性能。大量实验验证了其有效性与潜力。特别地,在挑战性的ImageNet-1k数据集上,4次示例微调下,平均FPR95降低5.17%,甚至超越此前方法在16次示例下的表现。代码已开源:https://github.com/AuroraZengfh/Local-Prompt。

原文摘要 · Abstract (English)

Out-of-Distribution (OOD) detection, aiming to distinguish outliers from known categories, has gained prominence in practical scenarios. Recently, the advent of vision-language models (VLM) has heightened interest in enhancing OOD detection for VLM through few-shot tuning. However, existing methods mainly focus on optimizing global prompts, ignoring refined utilization of local information with regard to outliers. Motivated by this, we freeze global prompts and introduce Local-Prompt, a novel coarse-to-fine tuning paradigm to emphasize regional enhancement with local prompts. Our method comprises two integral components: global prompt guided negative augmentation and local prompt enhanced regional regularization. The former utilizes frozen, coarse global prompts as guiding cues to incorporate negative augmentation, thereby leveraging local outlier knowledge. The latter employs trainable local prompts and a regional regularization to capture local information effectively, aiding in outlier identification. We also propose regional-related metric to empower the enrichment of OOD detection. Moreover, since our approach explores enhancing local prompts only, it can be seamlessly integrated with trained global prompts during inference to boost the performance. Comprehensive experiments demonstrate the effectiveness and potential of our method. Notably, our method reduces average FPR95 by 5.17% against state-of-the-art method in 4-shot tuning on challenging ImageNet-1k dataset, even outperforming 16-shot results of previous methods. Code is released at https://github.com/AuroraZengfh/Local-Prompt.

OOD检测视觉语言模型少样本学习局部提示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。