用低秩补偿器恢复4比特以下量化模型的精度损失。
MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators
- 引入低秩补偿器组，仅少量内存开销即可修复极端量化精度损失。
- 在3比特量化下实现显著加速，且无需校准数据，通用性强。
- 适配混合稀疏架构特性，支持跨模型和数据集迁移使用。
高效部署参数量巨大的混合专家（MoE）模型的关键在于量化。然而，当前先进MoE模型在极低比特量化（如低于4位）时仍存在明显精度下降。为此，我们提出MiLo，一种通过添加低秩补偿器组来增强高度量化MoE的新方法。这些补偿器仅消耗少量额外内存，却能显著恢复极端量化带来的精度损失。MiLo发现MoE模型因混合密集-稀疏结构在权重分布上具有独特特征，并采用自适应秩选择策略与迭代优化以缩小精度差距。该方法不依赖校准数据，可泛化至不同MoE模型与数据集而不过拟合。为避免3比特等极端量化带来的硬件效率问题，MiLo设计了适配Tensor Core的3比特内核，在3比特量化MoE模型上实现实测延迟加速。评估表明，MiLo在多种任务中均优于现有方法，适用于主流先进MoE模型。
原文摘要 · Abstract (English)
A critical approach for efficiently deploying Mixture-of-Experts (MoE) models with massive parameters is quantization. However, state-of-the-art MoE models suffer from non-negligible accuracy loss with extreme quantization, such as under 4 bits. To address this, we introduce MiLo, a novel method that augments highly quantized MoEs with a mixture of low-rank compensators. These compensators consume only a small amount of additional memory but significantly recover accuracy loss from extreme quantization. MiLo also identifies that MoEmodels exhibit distinctive characteristics across weights due to their hybrid dense-sparse architectures, and employs adaptive rank selection policies along with iterative optimizations to close the accuracy gap. MiLo does not rely on calibration data, allowing it to generalize to different MoE models and datasets without overfitting to a calibration set. To avoid the hardware inefficiencies of extreme quantization, such as 3-bit, MiLo develops Tensor Core-friendly 3-bit kernels, enabling measured latency speedups on 3-bit quantized MoE models. Our evaluation shows that MiLo outperforms existing methods on SoTA MoE models across various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。