让视觉提示随输入变化,提升模型表现。
Visual Instance-aware Prompt Tuning
- 根据每个输入生成个性化提示,融合全局提示
- 34个数据集测试均优于现有方法,准确率显著提升
- 参数量少于深度提示法,适合资源受限场景
视觉提示调优(VPT)作为视觉变换器的高效微调范式,传统方法使用对所有输入一致的数据集级提示,导致下游数据集差异大时性能不佳。本文提出视觉实例感知提示调优(ViaPT),基于每个输入生成实例级提示,并与数据集级提示融合,利用主成分分析(PCA)保留关键提示信息。我们发现VPT-Deep和VPT-Shallow代表两种极端情况,无法有效捕捉实例特异性信息;而随机降维仅能取得中间性能。相比之下,ViaPT在保留数据集知识的同时平衡实例信息,且相比VPT-Deep减少可学习参数。在34个多样化数据集上的实验表明,该方法持续优于当前最优基线,为视觉提示的分析与优化提供了新范式。
原文摘要 · Abstract (English)
Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. We observe that this strategy results in sub-optimal performance due to high variance in downstream datasets. To address this challenge, we propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input and fuses them with dataset-level prompts, leveraging Principal Component Analysis (PCA) to retain important prompting information. Moreover, we reveal that VPT-Deep and VPT-Shallow represent two corner cases based on a conceptual understanding, in which they fail to effectively capture instance-specific information, while random dimension reduction on prompts only yields performance between the two extremes. Instead, ViaPT overcomes these limitations by balancing dataset-level and instance-level knowledge, while reducing the amount of learnable parameters compared to VPT-Deep. Extensive experiments across 34 diverse datasets demonstrate that our method consistently outperforms state-of-the-art baselines, establishing a new paradigm for analyzing and optimizing visual prompts for vision transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。