让视觉状态空间模型仅用1个隐层维度,仍保持高精度并提速超1倍。
V"Mean"ba: Visual State Space Models only need 1 hidden dimension
- 用通道均值压缩隐藏状态,消除冗余维度
- 图像分类与分割任务中提速1.12倍,损失<3%准确率
- 无需训练,适配资源受限设备部署
视觉变换器在图像处理任务中表现优异,但自注意力的二次复杂度限制了其可扩展性及在资源受限设备上的部署。状态空间模型(SSMs)通过线性递归机制将序列建模复杂度从二次降低到线性,近年来被拓展至高分辨率视觉任务。然而,线性递归机制难以充分利用现代硬件的矩阵乘法单元,形成计算瓶颈。本文提出 extit{VMeanba},一种无需训练的压缩方法,通过均值操作消除SSM中的通道维度。关键观察是SSM块输出激活在通道间方差极低。利用此特性,对通道维度进行平均以降低计算开销,同时保持精度。在图像分类和语义分割任务上的评估表明, extit{VMeanba}实现最高1.12倍加速,准确率损失小于3%;与40%非结构化剪枝结合后,准确率下降仍低于3%。
原文摘要 · Abstract (English)
Vision transformers dominate image processing tasks due to their superior performance. However, the quadratic complexity of self-attention limits the scalability of these systems and their deployment on resource-constrained devices. State Space Models (SSMs) have emerged as a solution by introducing a linear recurrence mechanism, which reduces the complexity of sequence modeling from quadratic to linear. Recently, SSMs have been extended to high-resolution vision tasks. Nonetheless, the linear recurrence mechanism struggles to fully utilize matrix multiplication units on modern hardware, resulting in a computational bottleneck. We address this issue by introducing \textit{VMeanba}, a training-free compression method that eliminates the channel dimension in SSMs using mean operations. Our key observation is that the output activations of SSM blocks exhibit low variances across channels. Our \textit{VMeanba} leverages this property to optimize computation by averaging activation maps across the channel to reduce the computational overhead without compromising accuracy. Evaluations on image classification and semantic segmentation tasks demonstrate that \textit{VMeanba} achieves up to a 1.12x speedup with less than a 3\% accuracy loss. When combined with 40\% unstructured pruning, the accuracy drop remains under 3\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。