用脉冲神经网络生成稀疏低秩提示,提升视觉模型适配效率。
Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorization

- 通过低秩分解与脉冲神经网络结合,实现动态稀疏提示生成。
- 在多个基准上表现媲美主流方法,参数量更少。
- 适合资源受限场景下的视觉模型快速适配。
视觉提示(Visual Prompting, VP)通过在输入层加入可学习提示,高效适应大规模预训练视觉模型至下游任务。然而现有方法多采用密集像素级提示,存在冗余扰动、泛化能力弱和能效低的问题。为此,本文提出低秩脉冲提示(LoRSP),将类脑脉冲学习引入视觉提示任务。受脉冲神经元以离散脉冲信号传递信息、输出稀疏的启发,LoRSP 利用低秩因子分解构建提示子空间,并通过脉冲神经网络的积分-放电机制生成每实例特有的稀疏像素级提示。该设计在保持低秩约束的同时实现动态选择性提示,显著提升适应的紧凑性与鲁棒性。在五个异构视觉骨干网络及多个基准上的实验证明,LoRSP 在性能相当的前提下,所需可调参数更少。
原文摘要 · Abstract (English)
Visual Prompting (VP) has emerged as an efficient paradigm for adapting large-scale pre-trained vision models to downstream tasks by incorporating learnable prompts at the input level. However, existing VP methods typically employ dense pixel-level prompts, which often suffer from redundant perturbations, limited generalization and energy inefficiency. To overcome these limitations, we propose to integrate brain-inspired spiking learning into visual prompt learning tasks. As we know that spiking neuron can perform inexpensive information processing by transmitting the input data into discrete spike trains and return sparse outputs. Inspired by this, we propose \textbf{Lo}w-\textbf{R}ank visual \textbf{S}pike \textbf{P}rompting (LoRSP), a novel framework that learns dynamic low-rank sparse visual prompts naturally via a Spiking neuron learning mechanism. The core idea of LoRSP is to exploit the brain-inspired sparse firing mechanism of spiking neurons to generate pixel-level sparse prompt for each instance. To be specific, we first construct a series of prompt factors via low-rank factorization to capture distinct prompt subspaces. These prompt factors are then fed into an SNN architecture, which performs the integrate-and-fire process to emit spikes. As a result, our LoRSP generates a \emph{sparse} visual prompt while maintaining the low-rank constraint. This design enables instance-specific selective prompting, leading to more compact and robust adaptation across diverse downstream tasks. Extensive experiments on five heterogeneous vision backbones and multiple benchmarks demonstrate that LoRSP achieves competitive performance while requiring fewer tunable parameters compared to existing VP methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。