通过分层语义提示提升视觉模型微调效率与精度
Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuning
- 构建语义分层结构，区分共享与独立提示以学习层次化表征
- 在VTAB-1k上比VPT提升4.9%准确率，且推理成本更低
- 适合追求高效微调的视觉模型研究者和工业应用开发者
随着视觉模型规模持续扩大，视觉提示微调（VPT）作为一种参数高效迁移学习技术脱颖而出，其性能优于全量微调。然而，对每一层无差别应用提示会引发显著干扰，影响迁移能力；同时，VPT破坏原始自注意力结构，削弱视觉特征聚合能力，并缺乏显式挖掘判别性特征的机制。为此，我们提出语义分层提示（SHIP）微调策略：自适应构建语义层次，使用语义独立与共享提示学习层次化表示；引入属性提示与提示匹配损失增强特征判别性；采用解耦注意力机制提升鲁棒性并降低推理开销。实验表明，SHIP在ViT-B/16骨干网络上于VTAB-1k任务中较VPT提升4.9%准确率。代码已开源。
原文摘要 · Abstract (English)
As the scale of vision models continues to grow, Visual Prompt Tuning (VPT) has emerged as a parameter-efficient transfer learning technique, noted for its superior performance compared to full fine-tuning. However, indiscriminately applying prompts to every layer without considering their inherent correlations, can cause significant disturbances, leading to suboptimal transferability. Additionally, VPT disrupts the original self-attention structure, affecting the aggregation of visual features, and lacks a mechanism for explicitly mining discriminative visual features, which are crucial for classification. To address these issues, we propose a Semantic Hierarchical Prompt (SHIP) fine-tuning strategy. We adaptively construct semantic hierarchies and use semantic-independent and semantic-shared prompts to learn hierarchical representations. We also integrate attribute prompts and a prompt matching loss to enhance feature discrimination and employ decoupled attention for robustness and reduced inference costs. SHIP significantly improves performance, achieving a 4.9% gain in accuracy over VPT with a ViT-B/16 backbone on VTAB-1k tasks. Our code is available at https://github.com/haoweiz23/SHIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。