提出双提示协作框架,解决视觉语言模型微调时基础类与新类性能互损问题。
DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models
- 分离基础与新类的提示优化路径,通过双提示独立控制训练方向。
- 在多个骨干网络上提升基础类性能,同时保持对新类的泛化能力。
- 适合需要兼顾已知与未知类别表现的视觉语言模型微调场景。
CLIP-based提示微调在优化过程中普遍存在基础-新类权衡(BNT)问题,即对基础类持续微调会导致对新类(未见类)泛化能力下降。现有方法通过添加约束来调节提示微调过程以平衡该权衡,但这些约束作用于同一目标提示,无法彻底缓解基础类与新类优化方向的冲突。为此,我们提出无需修改主干的双提示协作(DPC)框架,首次在提示层面解耦基础与新类的优化过程。具体地,基于主提示克隆出一个可学习的并行提示,并引入可变权重解耦机制,独立控制双提示针对基础或新类的优化方向,从而避免泛化冲突。同时,提出动态硬负样本优化器,利用双提示构建更具挑战性的基础类优化任务以增强性能。为解释性,我们证明了提示向量在优化过程中特征通道不变性,为DPC的权重解耦提供理论支持。大量实验表明,DPC可在不引入额外知识的前提下显著提升基础类性能,同时维持对新类的泛化能力。代码已开源:https://github.com/JREion/DPC。
原文摘要 · Abstract (English)
The Base-New Trade-off (BNT) problem universally exists during the optimization of CLIP-based prompt tuning, where continuous fine-tuning on base (target) classes leads to a simultaneous decrease of generalization ability on new (unseen) classes. Existing approaches attempt to regulate the prompt tuning process to balance BNT by appending constraints. However, imposed on the same target prompt, these constraints fail to fully avert the mutual exclusivity between the optimization directions for base and new. As a novel solution to this challenge, we propose the plug-and-play Dual-Prompt Collaboration (DPC) framework, the first that decoupling the optimization processes of base and new tasks at the prompt level. Specifically, we clone a learnable parallel prompt based on the backbone prompt, and introduce a variable Weighting-Decoupling framework to independently control the optimization directions of dual prompts specific to base or new tasks, thus avoiding the conflict in generalization. Meanwhile, we propose a Dynamic Hard Negative Optimizer, utilizing dual prompts to construct a more challenging optimization task on base classes for enhancement. For interpretability, we prove the feature channel invariance of the prompt vector during the optimization process, providing theoretical support for the Weighting-Decoupling of DPC. Extensive experiments on multiple backbones demonstrate that DPC can significantly improve base performance without introducing any external knowledge beyond the base classes, while maintaining generalization to new classes. Code is available at: https://github.com/JREion/DPC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。