让视觉语言模型学会识别并纠正自身混淆错误,提升细粒度分类能力。
CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment
- 构建混淆库,显式建模类别间持续性混淆关系。
- 通过语义与样本双路径挖掘,解决50.72%的混淆样本对。
- 适合需要提升细粒度分类性能的研究者和应用开发者。
如CLIP等视觉语言模型在跨模态表征学习上取得显著进展,但仍存在视觉与语义相似类别间的系统性误分类问题。我们发现此类混淆模式并非随机,而是持续出现在特定类别对之间,揭示了模型内在偏见和有限的细粒度判别能力。为此,提出CAPT框架,使模型能够从自身的错配中学习。具体而言,构建混淆库以显式建模跨类别的稳定混淆关系;引入语义混淆挖掘器(SEM),通过语义差异与共性提示捕捉全局类别混淆;设计样本混淆挖掘器(SAM),利用差分适配器整合全局与局部上下文,检索代表性误分类样本;进一步设计多粒度差异专家(MGDE)模块,统一不同粒度的混淆信息,实现更鲁棒的混淆感知推理。在11个基准数据集上的大量实验表明,该方法显著降低由混淆引发的错误,增强基类与新类的判别力与泛化能力,成功解决50.72%的混淆样本对。代码将发布于https://github.com/greatest-gourmet/CAPT。
原文摘要 · Abstract (English)
Vision-language models like CLIP have achieved remarkable progress in cross-modal representation learning, yet suffer from systematic misclassifications among visually and semantically similar categories. We observe that such confusion patterns are not random but persistently occur between specific category pairs, revealing the model's intrinsic bias and limited fine-grained discriminative ability. To address this, we propose CAPT, a Confusion-Aware Prompt Tuning framework that enables models to learn from their own misalignment. Specifically, we construct a Confusion Bank to explicitly model stable confusion relationships across categories and misclassified samples. On this basis, we introduce a Semantic Confusion Miner (SEM) to capture global inter-class confusion through semantic difference and commonality prompts, and a Sample Confusion Miner (SAM) to retrieve representative misclassified instances from the bank and capture sample-level cues through a Diff-Manner Adapter that integrates global and local contexts. To further unify confusion information across different granularities, a Multi-Granularity Difference Expert (MGDE) module is designed to jointly leverage semantic- and sample-level experts for more robust confusion-aware reasoning. Extensive experiments on 11 benchmark datasets demonstrate that our method significantly reduces confusion-induced errors while enhancing the discriminability and generalization of both base and novel classes, successfully resolving 50.72 percent of confusable sample pairs. Code will be released at https://github.com/greatest-gourmet/CAPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。