用正负提示增强视觉模型,提升异常图像识别能力
Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models
- 通过大模型生成正负提示,聚焦类别边界特征
- 在CIFAR-100和ImageNet-1K上超越现有方法
- 适合需要高鲁棒性检测的视觉系统开发者
分布外(OOD)检测旨在划分内分布(ID)与外分布图像的分类边界。近年来,视觉语言模型(VLMs)通过融合视觉与文本模态,显著提升了OOD检测性能。现有方法引入负提示以强调图像特征与提示内容的差异,但负提示常包含大量非ID特征,导致重叠或误导信息影响效果。为此,我们提出正负提示监督机制,促使负提示捕捉跨类别特征,并将语义知识迁移至视觉模态以增强检测性能。方法基于大语言模型(LLMs)初始化类特定正负提示,随后优化:正提示聚焦类内特征,负提示突出类别边界特征。进一步采用图结构聚合优化后的提示表示,传播语义感知监督至视觉分支,提升基于能量的OOD检测器表现。在两个基准数据集CIFAR-100和ImageNet-1K上,覆盖八个不同OOD数据集及五种不同LLMs的实验表明,本方法优于当前最优基线。
原文摘要 · Abstract (English)
Out-of-distribution (OOD) detection is committed to delineating the classification boundaries between in-distribution (ID) and OOD images. Recent advances in vision-language models (VLMs) have demonstrated remarkable OOD detection performance by integrating both visual and textual modalities. In this context, negative prompts are introduced to emphasize the dissimilarity between image features and prompt content. However, these prompts often include a broad range of non-ID features, which may result in suboptimal outcomes due to the capture of overlapping or misleading information. To address this issue, we propose Positive and Negative Prompt Supervision, which encourages negative prompts to capture inter-class features and transfers this semantic knowledge to the visual modality to enhance OOD detection performance. Our method begins with class-specific positive and negative prompts initialized by large language models (LLMs). These prompts are subsequently optimized, with positive prompts focusing on features within each class, while negative prompts highlight features around category boundaries. Additionally, a graph-based architecture is employed to aggregate semantic-aware supervision from the optimized prompt representations and propagate it to the visual branch, thereby enhancing the performance of the energy-based OOD detector. Extensive experiments on two benchmarks, CIFAR-100 and ImageNet-1K, across eight OOD datasets and five different LLMs, demonstrate that our method outperforms state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。