arXiv:2507.19847cs.CV2025-07中稿 · ACMMM 2025被引 6

提升视觉语言模型在未知数据上的检测能力,同时避免性能下降。

Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection

  • 通过分离正负特征空间增强识别能力
  • 少样本训练下将误报率降低5.44%
  • 适合需要鲁棒性检测的实用场景

分布外(OOD)检测对构建可靠机器学习模型至关重要。尽管负提示调优提升了视觉语言模型的OOD检测能力,但这些模型在未见类别和风格上的泛化性能常会下降。为此,我们提出一种新方法——知识正则化负特征调优(KR-NFT),结合创新的负特征调优(NFT)架构与知识正则化(KR)优化策略。NFT对预训练文本特征施加分布感知变换,有效将正负特征分至不同空间,最大化区分分布内(ID)与分布外图像。此外,通过轻量级元网络引入图像条件可学习因子,实现对单个图像的动态适应,缓解对类别与风格变化的敏感性。相比传统负提示调优,NFT具备更优效率与可扩展性。为优化该架构,设计了KR策略,在增强ID与OOD判别力的同时,减轻预训练知识遗忘问题。该方法在已训练的ID类别上提升检测性能,同时改善对未见ID数据集的检测效果。值得注意的是,仅用少量ImageNet样本训练时,KR-NFT不仅提高分类准确率与检测能力,还在未探索的泛化设置下使FPR95降低5.44%。

原文摘要 · Abstract (English)

Out-of-distribution (OOD) detection is crucial for building reliable machine learning models. Although negative prompt tuning has enhanced the OOD detection capabilities of vision-language models, these tuned models often suffer from reduced generalization performance on unseen classes and styles. To address this challenge, we propose a novel method called Knowledge Regularized Negative Feature Tuning (KR-NFT), which integrates an innovative adaptation architecture termed Negative Feature Tuning (NFT) and a corresponding knowledge-regularization (KR) optimization strategy. Specifically, NFT applies distribution-aware transformations to pre-trained text features, effectively separating positive and negative features into distinct spaces. This separation maximizes the distinction between in-distribution (ID) and OOD images. Additionally, we introduce image-conditional learnable factors through a lightweight meta-network, enabling dynamic adaptation to individual images and mitigating sensitivity to class and style shifts. Compared to traditional negative prompt tuning, NFT demonstrates superior efficiency and scalability. To optimize this adaptation architecture, the KR optimization strategy is designed to enhance the discrimination between ID and OOD sets while mitigating pre-trained knowledge forgetting. This enhances OOD detection performance on trained ID classes while simultaneously improving OOD detection on unseen ID datasets. Notably, when trained with few-shot samples from ImageNet dataset, KR-NFT not only improves ID classification accuracy and OOD detection but also significantly reduces the FPR95 by 5.44\% under an unexplored generalization setting with unseen ID categories. Codes can be found at \href{https://github.com/ZhuWenjie98/KRNFT}.

OOD检测视觉语言模型负提示调优少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。