arXiv:2511.15015cs.PFcs.AI2025-11被引 7

动态调整专家精度,让大模型在单显卡上更快更省内存地运行。

Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference

  • 根据运行时流量动态分配专家精度,热专家保高精度,冷专家用低精度
  • 在相同显存下,相比固定精度量化,准确率提升4.48个百分点
  • 适合部署大模型的单卡推理场景,尤其适合显存受限的生产环境

混合专家(MoE)架构可在保持每标记计算量适中的前提下扩展大语言模型容量,但将MoE模型部署在单个内存受限的GPU上仍具挑战性,因专家权重占据主要显存。现有专家卸载和预取系统虽减少驻留集,但在激活密集时仍需在关键路径上承担加载开销。后训练量化(PTQ)可避免数据传输,但现有方法固定专家位宽且假设路由稳定,而实际上专家利用率呈长尾分布,热专家集可能随负载变化。我们提出DynaExq,一个面向单卡显存约束的实时混合精度推理系统。核心思想是:让主导运行时流量的专家以高精度常驻,其余专家则保留低精度备用,从而降低传输量并避免卸载/预取在密集激活下的等待延迟。DynaExq通过路由器日志预测长期专家热度,按预算可行的前n名规则选择每层高精度驻留集,并通过稳定专家句柄异步执行升/降级,确保前向传播始终使用完整材料化版本。在Qwen3-MoE-30B/80B及六个基准测试中,DynaExq在与静态PTQ相当的设备内存预算下,使Qwen3-80B准确率从73.09%提升至77.57%,且在批量大小为32时,吞吐量最高达卸载/预取基线的2.73倍。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) has become a practical architecture for scaling LLM capacity while keeping per-token compute modest, but deploying MoE models on a single, memory-limited GPU remains difficult because expert weights dominate the HBM footprint. Existing expert offloading and prefetching systems reduce the resident set, yet they often pay expert-loading costs on the critical path when activation becomes dense. Post-training quantization (PTQ) lowers the footprint without transfers, but prevailing pipelines fix expert bit-widths offline and assume routing remains stable, even though MoE expert utilization is heavy-tailed and the hot set can shift across workloads. We present DynaExq, a runtime-aware mixed-precision serving system that treats single-GPU MoE inference under a hard HBM envelope as an online, budget-constrained precision allocation problem. The key insight is to keep the experts that dominate runtime traffic resident at higher precision, while maintaining a low-precision fallback for the remaining experts, so the system can reduce transfer volume and avoid the waiting latency that limits offloading and prefetching under dense activation. DynaExq estimates long-horizon expert hotness from router traces, selects a per-layer high-precision resident set via a budget-feasible top-$n$ rule, and applies promotions and demotions asynchronously through stable expert handles so the forward pass always executes on a fully materialized expert version. Across Qwen3-MoE-30B/80B and six benchmarks, DynaExq improves accuracy over static PTQ on Qwen3-80B (73.09% to 77.57%) under comparable device-memory budgets and achieves up to 2.73x higher throughput than offloading/prefetch baselines at batch size 32.

MoE量化推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。