通过嵌入选择与子空间扩展，提升自监督模型的视觉提示初始化效果。
VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
- 基于嵌入空间中语义信息区域选择提示，实现精准初始化。
- 在小样本和复杂任务上显著提升性能，优于现有方法。
- 仅需一次前向传播，适合资源受限场景使用。
在大规模基础模型时代，对每个下游任务全量微调预训练网络往往资源开销过大。提示调优通过引入可训练提示而冻结主干网络，提供轻量化替代方案。然而，现有视觉提示调优方法常难以专化提示或丰富表示空间，尤其在自监督主干网络上表现更差。本文表明，这些局限在挑战性任务和数据稀缺场景下尤为明显，而正是这些情况下有效适应最为关键。为此，我们提出VIPAMIN，一种视觉提示初始化策略，通过（1）将提示对齐到嵌入空间中语义信息丰富的区域，（2）注入超越预训练子空间的新表示方向，从而增强自监督模型的适应能力。尽管方法简单——仅需一次前向传播和轻量级操作——但其在多种任务和数据规模下均持续提升性能，刷新了视觉提示调优的最新纪录。代码已开源：https://github.com/iamjaekyun/vipamin。
原文摘要 · Abstract (English)
In the era of large-scale foundation models, fully fine-tuning pretrained networks for each downstream task is often prohibitively resource-intensive. Prompt tuning offers a lightweight alternative by introducing tunable prompts while keeping the backbone frozen. However, existing visual prompt tuning methods often fail to specialize the prompts or enrich the representation space--especially when applied to self-supervised backbones. We show that these limitations become especially pronounced in challenging tasks and data-scarce settings, where effective adaptation is most critical. In this work, we introduce VIPAMIN, a visual prompt initialization strategy that enhances adaptation of self-supervised models by (1) aligning prompts with semantically informative regions in the embedding space, and (2) injecting novel representational directions beyond the pretrained subspace. Despite its simplicity--requiring only a single forward pass and lightweight operations--VIPAMIN consistently improves performance across diverse tasks and dataset sizes, setting a new state of the art in visual prompt tuning. Our code is available at https://github.com/iamjaekyun/vipamin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。