用预训练模型做三值量化，1020万词就能压缩并保持高性能。
Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models
- 从预训练模型出发，结合知识蒸馏与分组量化训练。
- 模型体积缩小3.61倍，零样本准确率达48.1%（7任务平均）。
- 发现三值量化新不稳定现象，适用于边缘部署研究者。
状态空间模型（SSMs）如Mamba-2虽具线性推理速度，但内存占用限制了边缘部署。以往三值化方法需从头训练1500亿词，本文表明预训练检查点已足够，使额外训练量降低1000倍。通过分组量化感知训练（QAT）结合冻结的FP16教师模型知识蒸馏，将13亿参数的Mamba-2压缩至744MB（原2687MB，压缩3.61倍），仅用10200万词（4卡时长4小时，单张H100）即实现48.1%零样本准确率（7任务平均），逼近Bi-Mamba的48.4%（置信区间±0.9个百分点）。该预训练+QAT设置揭示了“零比例崩溃”这一新型不稳定性，源于可学习量化尺度，且在从头训练中不会出现。进一步表明，适用于Transformer的后处理修正策略对SSMs无效，因递归导致误差累积。结果证明：三值化SSM无需昂贵从头训练，预训练+知识蒸馏的量化训练是高效替代方案。
原文摘要 · Abstract (English)
State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment. Prior ternary SSM work (Slender-Mamba) trains from scratch on 150B tokens; we show a pretrained checkpoint suffices, reducing the marginal token budget by 1,000x. Using grouped quantization-aware training (QAT) with knowledge distillation from a frozen FP16 teacher, we compress Mamba-2 1.3B to 3.61x (2,687 to 744 MB) and achieve 48.1% zero-shot accuracy (7-task average) in just 102M tokens (4 GPU-hours, single H100) -- approaching Bi-Mamba's 48.4% (within +/-0.9pp CI). This QAT-from-pretrained setting reveals zero-ratio collapse, a novel instability caused by learnable quantization scales that does not arise in from-scratch training. We further show that post-hoc correction strategies effective for Transformers fail for SSMs due to error accumulation through the recurrence. These results demonstrate that ternary SSMs do not require expensive from-scratch training: QAT from pretrained checkpoints with KD is a data-efficient alternative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。