arXiv:2505.18053cs.CVcs.AI2025-05被引 6

提升视觉语言模型适配效率,让小参数量模型也能泛化到未知类别。

FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation

  • 共享软监督上下文并加速输入输出,减少重复推理开销。
  • 引入正负提示空间互学习,零样本性能提升2.2倍训练速度。
  • 适合追求高效且强泛化的视觉语言模型迁移任务。

提示学习作为一种参数高效的下游任务适配方法,被广泛用于视觉语言模型(VLMs)的微调。传统硬提示需领域知识与迭代优化,而软提示依赖特定任务标签,限制了对未见类别的泛化能力。近期基于蒸馏的提示学习虽通过大教师模型和无监督知识迁移提升了泛化性,但重复的教师模型在线推理牺牲了提示学习原有的训练效率优势。本文提出更快的蒸馏式提示学习(FDBPL),通过跨多个训练阶段共享软监督上下文并实现加速的输入输出处理,解决该问题。此外,FDBPL引入区域感知提示学习范式,构建双正负提示空间,充分挖掘随机裁剪区域中的多层级信息。提出基于相似性-差异性的正负空间互学习机制,使学生CLIP模型在识别正确语义的同时学会拒绝弱相关概念,从而提升零样本性能。不同于现有方法以牺牲参数效率换取泛化性,FDBPL同时保持参数效率与强下游泛化能力。在11个数据集上的全面评估显示,其在基础到新类泛化、跨数据集迁移及鲁棒性测试中表现优异,训练速度提升2.2倍。

原文摘要 · Abstract (English)

Prompt learning as a parameter-efficient method that has been widely adopted to adapt Vision-Language Models (VLMs) to downstream tasks. While hard-prompt design requires domain expertise and iterative optimization, soft-prompt methods rely heavily on task-specific hard labels, limiting their generalization to unseen categories. Recent popular distillation-based prompt learning methods improve generalization by exploiting larger teacher VLMs and unsupervised knowledge transfer, yet their repetitive teacher model online inference sacrifices the inherent training efficiency advantage of prompt learning. In this paper, we propose {\large {\textbf{F}}}aster {\large {\textbf{D}}}istillation-{\large {\textbf{B}}}ased {\large {\textbf{P}}}rompt {\large {\textbf{L}}}earning (\textbf{FDBPL}), which addresses these issues by sharing soft supervision contexts across multiple training stages and implementing accelerated I/O. Furthermore, FDBPL introduces a region-aware prompt learning paradigm with dual positive-negative prompt spaces to fully exploit randomly cropped regions that containing multi-level information. We propose a positive-negative space mutual learning mechanism based on similarity-difference learning, enabling student CLIP models to recognize correct semantics while learning to reject weakly related concepts, thereby improving zero-shot performance. Unlike existing distillation-based prompt learning methods that sacrifice parameter efficiency for generalization, FDBPL maintains dual advantages of parameter efficiency and strong downstream generalization. Comprehensive evaluations across 11 datasets demonstrate superior performance in base-to-new generalization, cross-dataset transfer, and robustness tests, achieving $2.2\times$ faster training speed.

提示学习视觉语言模型高效训练零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。