arXiv:2501.18936cs.LGcs.CV2025-01中稿 · ICLR被引 6

给视觉提示调优引入可自适应的专家,提升模型表达能力

Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts

  • 将提示专家视为可动态调整的MoE组件,增强表达力
  • 在VTAB-1K上比全微调高出7.34%,在FGVC上高1.04%
  • 参数量少于原方法,适合高效微调场景

视觉提示调优(VPT)通过插入可学习的提示标记,实现预训练视觉模型对下游任务的参数高效适配。尽管其在实践中表现良好,但理论理解仍不充分。基于最近揭示的混合专家(MoE)与提示方法之间的联系——每个注意力头可视为多个MoE模型的组合——我们重新将VPT解释为在这些MoE结构中引入新的提示专家。我们发现现有VPT框架的关键缺陷在于提示专家的功能表达受限,因其保持静态而适应性不足。为此,提出视觉自适应提示调优(VAPT),赋予提示专家更强的表达能力,同时保持参数效率。在VTAB-1K和FGVC上的实证评估显示,VAPT分别超越全微调基线7.34%和1.04%。此外,VAPT始终优于VPT,且所需额外参数更少。理论分析表明,VAPT实现了最优样本效率。这些结果共同证明了该方法的理论基础与实际优势。

原文摘要 · Abstract (English)

Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt tokens. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on the recently established connection between Mixture of Experts (MoE) and prompt-based methods, wherein each attention head can be conceptualized as a composition of multiple MoE models, we reinterpret VPT as the introduction of new prompt experts into these MoE structures. We identify a key limitation in existing VPT frameworks: the restricted functional expressiveness of prompt experts, which remain static and thus limited in their adaptability. To address this, we propose Visual Adaptive Prompt Tuning (VAPT), a novel method that endows prompt experts with enhanced expressiveness while preserving parameter efficiency. Empirical evaluations on VTAB-1K and FGVC demonstrate that VAPT achieves substantial performance improvements, surpassing fully fine-tuned baselines by 7.34% and 1.04%, respectively. Moreover, VAPT consistently outperforms VPT while requiring fewer additional parameters. Furthermore, our theoretical analysis indicates that VAPT achieves optimal sample efficiency. Collectively, these results underscore the theoretical grounding and empirical advantages of our approach.

视觉提示MoE参数高效自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。