HetRoute优化跨服务器MoE模型推理,降低延迟与流量。
HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
- 构建统一成本模型,融合传输、计算、排队和量化损耗
- 离线部署+在线整组路由,降低平均延迟59%、P99延迟58%
- 适合大规模分布式边缘MoE推理,兼顾性能与精度
混合专家(MoE)模型已成为大规模AI服务的主流架构,但在地理分布的异构边缘服务器上部署仍具挑战。当一个令牌激活的Top-k专家分布在多个服务器时,最优路由需综合考虑跨服务器带宽、异构GPU算力、GPU-CPU专家加载延迟、瞬时队列积压及副本级量化质量损失。现有方法分别处理这些因素,缺乏统一的在线多服务器协同路由框架。本文提出HetRoute,一种面向分布式边缘MoE推理的异构成本感知协同路由框架。它引入统一的每任务成本模型,显式建模四类成本:跨服务器传输、GPU-CPU卸载、带队列的GPU计算,以及量化引起的质量惩罚。离线阶段通过耦合路由成本的部署算法确定专家服务器位置、GPU-CPU驻留策略与副本精度;在线阶段通过精确枚举或束搜索整体最小化瓶颈层成本。理论分析证明了降级可行性、参与服务器数量上限、小候选域下逐层最优性及在线计算复杂度。基于三个MoE模型在10台异构边缘服务器上的追踪评估显示,HetRoute将平均推理延迟降低最高达59.0%,P99延迟降低58.0%,跨服务器流量减少72.1%,吞吐量提升2.13倍,同时将质量退化控制在配置预算内。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread across multiple servers, the optimal routing depends jointly on cross-server link bandwidth, heterogeneous GPU computing capability, GPU-CPU expert loading delay, instantaneous queueing backlog, and replica-level quantization quality loss. Existing distributed inference and MoE serving methods address these factors separately and do not provide a unified framework for online multi-server collaborative routing. In this paper, we propose HetRoute, a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference. HetRoute introduces a unified per-assignment cost model that explicitly captures four cost components: cross-server transmission, GPU-CPU offloading, GPU computation with queueing, and quantization-induced quality penalty. Guided by this model, the offline stage determines expert server placement, GPU-CPU residency, and replica precision through a routing-cost-coupled deployment algorithm, while the online stage routes the Top-k activated expert set as a whole by minimizing the bottleneck layer cost via exact enumeration or beam search. Theoretical analysis establishes fallback feasibility, a bound on the number of participating servers, per-layer optimality for small candidate domains, and online computational complexity. Trace-driven evaluation on three MoE models over a heterogeneous 10-server edge testbed shows that HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%, cuts cross-server traffic by up to 72.1%, and achieves 2.13x throughput improvement compared with representative baselines, while keeping quality degradation within the configured budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。