为视觉马尔可夫模型设计了针对性的提示方法,提升微调效果。
Selective Visual Prompting in Vision Mamba
- 提出分步激活门控的轻量级提示机制,适配马尔可夫结构特性。
- 在多个大规模数据集上优于现有方法,显著提升特征传播效率。
- 适合需要高效微调视觉马尔可夫模型的研究者使用。
预训练的视觉马尔可夫模型(Vim)因其独特的选择性状态空间设计,在计算高效的前提下表现出色。为进一步拓展其在多样化下游任务中的应用,可采用高效的微调技术——视觉提示。然而,现有提示方法主要针对基于全局注意力的视觉变压器(ViT)设计,忽略了Vim特有的逐标记压缩与传播特性。具体而言,前置提示标记无法有效激活整个序列的输入门和遗忘门,阻碍了判别信息的提取与传播。为此,本文提出一种专为Vim设计的新型选择性视觉提示(SVP)方法。SVP通过轻量级选择性提示器实现逐标记提示生成,确保马尔可夫块内更新门和遗忘门的自适应激活,促进判别信息传播。此外,考虑到Vim同时传播跨层共享信息与层内特异性信息,我们进一步引入双路径结构:跨提示(Cross-Prompting)使用跨层共享参数,内提示(Inner-Prompting)使用独立参数,分别促进共享与特异性信息的传播。大量实验结果表明,所提SVP在多个大规模基准上显著优于现有先进方法。
原文摘要 · Abstract (English)
Pre-trained Vision Mamba (Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their unique design of selective state space models. To further extend their applicability to diverse downstream vision tasks, Vim models can be adapted using the efficient fine-tuning technique known as visual prompting. However, existing visual prompting methods are predominantly tailored for Vision Transformer (ViT)-based models that leverage global attention, neglecting the distinctive sequential token-wise compression and propagation characteristics of Vim. Specifically, existing prompt tokens prefixed to the sequence are insufficient to effectively activate the input and forget gates across the entire sequence, hindering the extraction and propagation of discriminative information. To address this limitation, we introduce a novel Selective Visual Prompting (SVP) method specifically for the efficient fine-tuning of Vim. To prevent the loss of discriminative information during state space propagation, SVP employs lightweight selective prompters for token-wise prompt generation, ensuring adaptive activation of the update and forget gates within Mamba blocks to promote discriminative information propagation. Moreover, considering that Vim propagates both shared cross-layer information and specific inner-layer information, we further refine SVP with a dual-path structure: Cross-Prompting and Inner-Prompting. Cross-Prompting utilizes shared parameters across layers, while Inner-Prompting employs distinct parameters, promoting the propagation of both shared and specific information, respectively. Extensive experimental results on various large-scale benchmarks demonstrate that our proposed SVP significantly outperforms state-of-the-art methods. Our code is available at https://github.com/zhoujiahuan1991/AAAI2025-SVP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。