用视觉语义增强提示调优，仅改0.74%参数就提升图像分类性能
Adapting Vision Foundation Models with Cascaded Semantics
- 在输入和特征空间注入颜色纹理形状等先验知识
- 34个数据集上仅调0.74%参数即达更优效果
- 适合追求高效微调的视觉模型研究者
提示调优是自然语言处理中一种主流的参数高效适配范式，最近被拓展至计算机视觉领域。视觉提示调优（VPT）通过更新少量额外的提示参数来适配预训练的视觉变压器（ViTs）。然而，现有的视觉提示随机初始化，未利用如自然语言中的指令等先验知识。为此，我们向VPT中注入两种互补的语义先验：通过经典手工算子提取的颜色、纹理、形状等基础图像先验，注入输入空间；自注意力图则在特征空间提供实例感知语义。我们进一步提出级联方案，将两类先验贯穿于ViT适配全过程。在34个具有挑战性的图像分类数据集上的实验表明，仅调整0.74%的ViT参数即可实现优越的下游适配性能。
原文摘要 · Abstract (English)
Prompt tuning, a leading parameter-efficient adaptation paradigm in NLP, has recently been extended to computer vision. Visual prompt tuning (VPT) adapts pre-trained vision transformers (ViTs) by updating a small set of additional prompt parameters. However, existing visual prompts are randomly initialized and do not exploit prior knowledge, such as instructions in NLP. We address this gap by injecting two complementary semantic priors into VPT. Fundamental image priors, including color, texture, and shape, are extracted with classical hand-crafted operators and injected into the input space, while self-attention maps provide instance-aware semantics in the feature space. We further propose a cascaded scheme that integrates both priors throughout ViT adaptation. Experiments on 34 challenging image classification datasets demonstrate superior downstream adaptation while tuning only 0.74% of ViT parameters. Project page: https://xixiaouab.github.io/Cascaded-Semantics/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。