针对视觉Mamba量化难题，提出快速高效且精度几乎无损的后训练量化方法。
PTQ4VM: Post-Training Quantization for Visual Mamba
- 设计分令牌静态量化与平滑尺度联合学习策略应对量化挑战
- 在多个任务上实现高达1.83倍推理加速，精度损失可忽略
- 首个专门针对视觉Mamba的量化研究，适用于多种骨干网络
视觉Mamba将选择性状态空间模型Mamba扩展至视觉任务，通过固定顺序逐个处理图像标记并累积信息生成输出。尽管其在多种任务中以低计算成本实现高质量输出而日益流行，但对量化极为敏感，进一步优化困难。分析表明，固定标记访问顺序引入三大量化问题：标记级方差、通道级异常值及激活值长尾分布。为此，本文提出面向视觉Mamba的后训练量化方法PTQ4VM，引入两项核心策略：分令牌静态（PTS）量化和平滑尺度与步长联合学习（JLSS）。据我们所知，这是首个针对视觉Mamba的量化研究。PTQ4VM可在15分钟内将预训练模型转换为量化格式，且性能无明显下降。大规模分类与回归任务实验验证其有效性，在GPU上实现最高1.83倍加速，相比FP16精度损失极小。代码已开源。
原文摘要 · Abstract (English)
Visual Mamba is an approach that extends the selective space state model, Mamba, to vision tasks. It processes image tokens sequentially in a fixed order, accumulating information to generate outputs. Despite its growing popularity for delivering high-quality outputs at a low computational cost across various tasks, Visual Mamba is highly susceptible to quantization, which makes further performance improvements challenging. Our analysis reveals that the fixed token access order in Visual Mamba introduces unique quantization challenges, which we categorize into three main issues: 1) token-wise variance, 2) channel-wise outliers, and 3) a long tail of activations. To address these challenges, we propose Post-Training Quantization for Visual Mamba (PTQ4VM), which introduces two key strategies: Per-Token Static (PTS) quantization and Joint Learning of Smoothing Scale and Step Size (JLSS). To the our best knowledge, this is the first quantization study on Visual Mamba. PTQ4VM can be applied to various Visual Mamba backbones, converting the pretrained model to a quantized format in under 15 minutes without notable quality degradation. Extensive experiments on large-scale classification and regression tasks demonstrate its effectiveness, achieving up to 1.83x speedup on GPUs with negligible accuracy loss compared to FP16. Our code is available at https://github.com/YoungHyun197/ptq4vm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。