arXiv:2503.09509cs.CV2025-03ICCV被引 2

针对视觉Mamba模型设计高效量化方法,实现低比特压缩同时保持高精度。

ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba

  • 提出快速凸组合优化算法,高效搜索最优码本。
  • 在3比特量化下保持90.1%准确率,优于现有方法。
  • 适合边缘设备部署,特别适用于视觉Mamba模型压缩。

视觉Mamba网络(ViMs)将选择性状态空间模型(Mamba)扩展至多种视觉任务,展现出巨大潜力。向量量化(VQ)作为一种高效的压缩技术,可将网络权重分解为码本和分配,显著降低内存占用和计算延迟,使ViMs可在边缘设备上部署。尽管现有VQ方法在卷积神经网络和基于Transformer的网络中已实现极低比特量化(如3比特、2比特、1比特),但直接应用于ViMs会导致精度不理想。我们识别出两大关键挑战:1)ViMs中Mamba模块的权重包含大量异常值,显著放大量化误差;2)现有最新VQ方法在应用于ViMs时存在内存开销大、校准时间长、码本搜索效果不佳等问题。本文提出ViM-VQ,一种专为ViMs设计的高效后训练向量量化方法。ViM-VQ包含两个创新组件:1)一种快速凸组合优化算法,高效更新凸组合与凸包以搜索最优码本;2)增量式向量量化策略,逐步确认最优码本以缓解截断误差。实验结果表明,ViM-VQ在多种视觉任务中均实现了低比特量化下的最先进性能。

原文摘要 · Abstract (English)

Visual Mamba networks (ViMs) extend the selective state space model (Mamba) to various vision tasks and demonstrate significant potential. As a promising compression technique, vector quantization (VQ) decomposes network weights into codebooks and assignments, significantly reducing memory usage and computational latency, thereby enabling the deployment of ViMs on edge devices. Although existing VQ methods have achieved extremely low-bit quantization (e.g., 3-bit, 2-bit, and 1-bit) in convolutional neural networks and Transformer-based networks, directly applying these methods to ViMs results in unsatisfactory accuracy. We identify several key challenges: 1) The weights of Mamba-based blocks in ViMs contain numerous outliers, significantly amplifying quantization errors. 2) When applied to ViMs, the latest VQ methods suffer from excessive memory consumption, lengthy calibration procedures, and suboptimal performance in the search for optimal codewords. In this paper, we propose ViM-VQ, an efficient post-training vector quantization method tailored for ViMs. ViM-VQ consists of two innovative components: 1) a fast convex combination optimization algorithm that efficiently updates both the convex combinations and the convex hulls to search for optimal codewords, and 2) an incremental vector quantization strategy that incrementally confirms optimal codewords to mitigate truncation errors. Experimental results demonstrate that ViM-VQ achieves state-of-the-art performance in low-bit quantization across various visual tasks.

视觉Mamba向量量化模型压缩边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。