arXiv:2506.23856cs.CV2025-06IJCV被引 16

用类别信息动态调整提示,解决视觉语言模型适应新任务时的性能下降问题。

A Closer Look at Conditional Prompt Tuning for Vision-Language Models

  • 用文本类别信息替代图像信息作为提示条件,提升模型泛化能力。
  • 在11个数据集上显著优于现有无条件提示调优方法,提升平均3.49%准确率。
  • 可作为插件使用,几乎不增加计算开销,适合快速部署到现有模型。

尽管提示调优(PT)在将大规模视觉-语言预训练模型(VLPMs)适配下游任务方面展现出巨大潜力,但其常面临基础-新任务权衡(BNT)困境:模型对基础任务越优化,对新任务的泛化能力越弱。现有条件提示调优通过用视觉图像信息(VII)条件化动态提示改善了部分泛化能力。本文发现,以VII为提示条件实际效果不佳,甚至随机噪声条件提示表现更优。进一步分析表明,基于文本类别信息(TCI)的动态提示学习才是解决BNT的关键。为此,我们提出类别自适应提示调优(CaPT),从基础类别中学习TCI条件化提示,实现对新类别的快速适配。值得注意的是,CaPT可作为插件用于已有无条件提示调优方案,有效缓解BNT问题。在11个数据集上的大量实验表明,CaPT持续提升5种强基线模型性能,且计算开销极低。结合我们此前提出的DePT框架,构建出新方法DeCaPT,其在11个数据集上的平均准确率较当前最优条件提示调优方案高出3.49%。

原文摘要 · Abstract (English)

Despite the great promise of Prompt Tuning (PT) in adapting large Vision-Language Pretrained Models (VLPMs) to downstream tasks, they often struggle to overcome the Base-New Tradeoff (BNT) dilemma: as VLPMs are better tuned to a base task, their ability to generalize to new tasks diminishes. Recent work on conditional PT addresses this problem by replacing static prompts with dynamic Visual Image Information (VII)-conditioned prompts, improving the model's generalization to new tasks to some extent. In this work, we first identify a critical issue with existing conditional PT methods: using VII as the "condition" of prompts yields suboptimal performance, and even random noise-conditioned prompts can outperform the VII-conditioned counterparts. On further analysis, we find that learning dynamic prompts conditioned on Textual Class Information (TCI) is the key to solving the BNT problem. Motivated by this, we then propose Class-adaptive Prompt Tuning (CaPT), which enables fast adaptation of tuned models to new classes by learning TCI-conditioned prompts from base classes. Remarkably, CaPT can be used as a plugin to mitigate the BNT problem for existing unconditional PT schemes. Extensive experiments on 11 datasets show that CaPT consistently improves the performance of five strong unconditional PT baselines with negligible additional computational cost. Additionally, by integrating CaPT with our recently proposed DePT framework, we devise a new conditional PT approach, termed DeCaPT, which outperforms the H ACC of the state-of-the-art conditional PT scheme by 3.49%, averaged over the 11 datasets. Code: https://github.com/Koorye/CaPT.

提示调优视觉语言模型类别自适应泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。