动态量化专家，让边缘设备高效运行MoE模型
DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge
- 根据专家重要性动态调整精度，实时优化计算资源
- 在真实设备上实现3.44倍到22.7倍的首令牌延迟降低
- 适合需要低延迟、高精度的边缘AI部署场景
尽管MoE模型具有计算效率优势，但多专家架构带来的巨大内存占用和输入输出开销，给资源受限的边缘平台实时推理带来挑战。现有静态方法难以平衡延迟与精度。我们发现专家重要性高度不均且随深度变化。基于此，提出DyMoE：一种面向边缘高效推理的动态混合精度量化框架。通过重要性感知优先级调度，动态量化运行时专家；深度自适应调度以保护关键层语义完整性；以及前瞻预取来重叠I/O等待。在商用边缘硬件上的实验表明，相比最先进卸载基线，DyMoE将首令牌延迟（TTFT）降低3.44倍至22.7倍，吞吐率提升高达14.58倍（TPOT），实现资源受限设备上实时、保精度的MoE推理。
原文摘要 · Abstract (English)
Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。