动态融合多源提示,提升视觉任务迁移效果
Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfer
- 通过可微分指标与正则化策略,动态学习最优提示权重
- 在VTAB基准上实现当前最佳性能,显著提升迁移效果
- 适合资源受限场景下的高效模型适配,尤其擅长多源知识融合
提示调优已成为在资源受限系统中适应基础模型到下游任务的轻量级策略。随着预训练提示成为宝贵资产,结合多个源提示可通过互补知识增强新任务的泛化能力。然而,简单聚合常忽略不同源提示对目标任务的贡献差异。为此,我们提出HGPrompt,一种动态框架,用于学习最优集成权重。该权重通过联合最大化信息论度量的可迁移性并最小化梯度冲突来优化。具体而言,我们设计了一个可微分的提示可迁移性度量,以捕捉提示诱导特征在目标任务上的区分能力。同时,HGPrompt基于海森矩阵和费雪信息量匹配不同源提示的梯度方差,确保稳定且一致的知识迁移,抑制它们之间的梯度冲突。在大规模VTAB基准上的大量实验验证了HGPrompt的先进性能,证明其在有效多源提示迁移中学习最优集成的有效性。
原文摘要 · Abstract (English)
Prompt tuning has emerged as a lightweight strategy for adapting foundation models to downstream tasks, particularly for resource-constrained systems. As pre-trained prompts become valuable assets, combining multiple source prompts offers a promising approach to enhance generalization for new tasks by leveraging complementary knowledge. However, naive aggregation often overlooks different source prompts have different contribution potential to the target task. To address this, we propose HGPrompt, a dynamic framework that learns optimal ensemble weights. These weights are optimized by jointly maximizing an information-theoretic metric for transferability and minimizing gradient conflicts via a novel regularization strategy. Specifically, we propose a differentiable prompt transferability metric to captures the discriminability of prompt-induced features on the target task. Meanwhile, HGPrompt match the gradient variances with respect to different source prompts based on Hessian and Fisher Information, ensuring stable and coherent knowledge transfer while suppressing gradient conflicts among them. Extensive experiments on the large-scale VTAB benchmark demonstrate the state-of-the-art performance of HGPrompt, validating its effectiveness in learning an optimal ensemble for effective multi-source prompt transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。