提出新量化方法，让专家模型路由更稳定、精度更高。
Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models
- 设计价值与结构对齐目标，保持专家选择一致性
- 在主流MoE模型上提升量化后性能，超越现有方法
- 无需额外计算开销，可无缝集成到现有框架
Mixture-of-Experts（MoE）模型通过仅激活部分专家实现高效扩展，但其大量参数仍需量化以支持实际部署。然而，与密集模型不同，MoE对路由不稳定性敏感：微小的量化扰动可能改变top-$k$专家选择，导致计算路径变化并降低模型质量。本文提出面向量值与结构对齐的量化方法（VSRAQ），一种专用于MoE的后训练量化目标，旨在保持量化前的专家选择行为。VSRAQ结合两个互补目标：价值对齐（匹配路由相关逻辑值或得分）与结构对齐（保持专家排序及top-$k$决策边界）。通过维持路由一致性，VSRAQ在不引入推理时开销的前提下显著减少量化带来的性能下降，并可融入现有量化框架。在多个近期MoE基础模型上的实验表明，该方法有效提升专家选择一致性，且持续优于仅重建和路由器感知的基线方法。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each token, but their large number of expert parameters still makes quantization essential for practical deployment. Unlike dense models, however, MoE models are sensitive to routing instability: small quantization-induced perturbations can change the top-$k$ expert selection, altering the computation path and degrading model quality. We propose Value-and-Structure Routing Alignment for Quantization (VSRAQ), a MoE-specific post-training quantization objective that preserves pre-quantization expert-selection behavior under quantization. VSRAQ combines two complementary objectives that jointly preserve expert-selection behavior: value alignment, which matches routing-relevant logits or scores, and structure alignment, which preserves expert ordering and top-$k$ decision boundaries. By maintaining routing consistency, VSRAQ reduces quantization-induced degradation without introducing any inference-time overhead and can be integrated into existing quantization frameworks. Experiments on recent MoE foundation models show that VSRAQ improves expert-selection consistency and consistently outperforms reconstruction-only and router-aware baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。