arXiv:2605.06055cs.DCcs.LG2026-05

移除中间缓存,直接读写专家窗口,加速昇腾平台MoE推理。

Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend

论文配图:Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend
图 1 · 摘自论文原文
  • 绕过中继缓冲区,直接在目标专家窗口放置与读取数据。
  • 预填充和解码阶段延迟均降低,首令牌时间减少18%以上。
  • 适合追求低延迟的在线服务部署,尤其适用于昇腾硬件。

Mixture-of-Experts(MoE)推理需跨设备大规模令牌交换,调度与合并成为预填充和解码阶段的主要瓶颈。除网络传输外,路由驱动的布局变换、临时中继和输出恢复也会引入显著开销。现有MoE通信路径多依赖缓冲区,使用显式进程间中继及集体传输周围的重排序缓冲区。本文提出一种面向昇腾系统的无中继缓冲通信设计,将调度与合并重构为直接放置至目标专家窗口并直接读取远程专家窗口。基于全局池化高带宽内存与对称内存分配,该设计移除了大部分中间中继与重排序缓冲区,仅保留轻量级控制状态(如计数、偏移和同步元数据)。我们实现了两种调度策略:面向吞吐的预填充调度,含更丰富的规划状态;面向低延迟的紧凑解码调度。在昇腾平台的MoE工作负载实验表明,两种场景下调度与合并延迟均下降。服务层面,实现提升首令牌时间(TTFT),保持竞争力的每输出令牌时间(TPOT),并在实际延迟约束下扩大可行调度空间。结果表明,在具备全局可寻址设备内存的平台上,减少专家执行过程中的中间缓冲与输出恢复是加速MoE推理的有效方向。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) inference requires large-scale token exchange across devices, making dispatch and combine major bottlenecks in both prefill and decode. Beyond network transfer, routing-driven layout transformation, temporary relay, and output restoration can add substantial overhead. Existing MoE communication paths are often buffer-centric, using explicit inter-process relay and reordering buffers around collective transfer. This report presents a relay-buffer-free communication design for MoE inference acceleration on Ascend systems. The design reorganizes dispatch and combine around direct placement into destination expert windows and direct reading from remote expert windows. Built on globally pooled high-bandwidth memory and symmetric-memory allocation, it removes most intermediate relay and reordering buffers while retaining only lightweight control state, including counts, offsets, and synchronization metadata. We instantiate the design as two schedules for the main phases of MoE inference: a prefill schedule with richer planning state for throughput-oriented execution, and a compact decode schedule for latency-sensitive execution. Experiments on Ascend-based MoE workloads show reduced dispatch and combine latency in both settings. At the serving level, the implementation improves time to first token (TTFT), preserves competitive time per output token (TPOT), and enlarges the feasible scheduling space under practical latency constraints. These results indicate that, on platforms with globally addressable device memory, reducing intermediate buffering and output restoration around expert execution is an effective direction for accelerating MoE inference.

MoE推理昇腾架构通信优化低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。