针对MoE芯片中专家负载不均问题，动态调度热专家存储位置以降低通信与内存压力。
HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference
- 根据专家热度和资源压力，动态决定热专家的存留与迁移。
- 在预填充和解码阶段分别降低43.6%和43.0%的端到端延迟。
- 适合大规模MoE芯片系统推理优化，尤其关注通信与内存瓶颈场景。
混合专家（MoE）大语言模型在推理时仅激活少数专家，但令牌路由导致专家负载严重不均：少数热专家持续接收大量令牌，其余专家负载极低。在3.5D多芯片系统中，这种不均不仅引发计算不平衡，还加剧了通信、内存带宽、I/O及执行队列的压力。因此核心问题并非单纯减少令牌移动，而是动态地将热专家副本放置并复用在不同内存层级。本文提出HCRMap，一种面向压力感知的热专家驻留映射框架，用于3.5D MoE芯片推理中的专家副本管理。基于专家热度、权重加载开销、迁移开销及运行时资源压力，HCRMap动态决策专家的提升、保留、降级或驱逐，并将路由令牌组映射到合适的驻留副本，从而协同缓解通信、内存与队列瓶颈。实验结果表明，与Hydra相比，预填充和解码阶段分别降低43.6%和43.0%的端到端延迟；相比MoEntwine，分别降低34.5%和33.1%；相比PIMoE，分别降低46.7%和46.0%。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded. On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues. Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers. This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted. It then maps routed token groups to suitable resident replicas, thereby jointly mitigating communication, memory, and queue bottlenecks. Experimental results show that HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in the prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46.0% over PIMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。