针对大模型推理的跨设备数据传输难题，提出感知网络拓扑的智能调度方案。
Topology-Aware Data Movement for Disaggregated GPU Inference
- 根据物理连接差异动态选择传输路径，利用NVLink、CXL等高速通道提升效率
- 通过分层流水传输隐藏76%至100%的延迟，部分场景下完全隐藏传输耗时
- 适用于大规模异构集群部署，尤其适合使用CXL内存扩展的高性能推理系统
拆分式大模型推理带来数据中心网络挑战：当预填充与解码在不同GPU池运行时，需在两者间传递KV缓存。以700亿参数模型为例，每请求需传输1.3 GB，生产规模下总带宽需求超100 GB/s。现有系统如DistServe、Splitwise和Mooncake均采用统一RDMA策略，忽视了实际带宽差异——同一域内通过NVLink 4.0可达900 GB/s（NVLink 5达1.8 TB/s，差距扩大至144倍），跨节点为InfiniBand约50 GB/s，跨数据中心仅TCP约12.5 GB/s。本文设计拓扑感知传输协调器，在启动时发现互联层级，并为每次传输选择最优路径。结合三种机制：(1) 分层流水传输，将传输与预填充计算重叠，根据传输方式隐藏76%~100%延迟，其中NVLink与PCIe传输可完全隐藏；(2) 针对混合专家模型的NVLink域感知放置，联合优化专家调度与KV缓存局部性；(3) 使用CXL 3.0内存扩展器作为共享溢出层，容量提升6倍，延迟仅为NVMe的1/86。完整评估需多节点异构集群及尚未普及的CXL 3.0硬件，超出学术资源范围。本文提供带宽建模、组件实现与三类架构投影分析，显示相较统一RDMA，传输延迟降低3至18倍。
原文摘要 · Abstract (English)
Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly. When prefill and decode run on separate GPU pools, the KV cache must be transferred between them. For a 70B model this is 1.3 GB per request, exceeding 100 GB/s aggregate at production scale. Yet DistServe, Splitwise, and Mooncake all use uniform RDMA, ignoring that bandwidth between two GPUs varies by 72x depending on their physical relationship: 900 GB/s via NVLink 4.0 within a domain (1.8 TB/s on NVLink 5, widening the gap to 144x), 50 GB/s via InfiniBand across nodes, 12.5 GB/s via TCP across data centers. We design a topology-aware transfer orchestrator that discovers interconnect hierarchy at startup and selects optimal transport per transfer. Three mechanisms work together: (1) pipelined layer-by-layer transfer that overlaps transmission with ongoing prefill, hiding 76 to 100 percent of transfer latency behind computation depending on transport, with NVLink and PCIe transfers hidden entirely; (2) NVLink domain-aware placement for Mixture-of-Experts models that co-optimizes expert dispatch with KV cache locality; and (3) CXL 3.0 memory expanders as a shared overflow tier providing 6x capacity at 86x lower latency than NVMe. Full evaluation requires multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware that is beyond academic resources and not yet available in GPU clouds. We present analytical bandwidth models, component implementations, and projected analysis across three architectures showing 3 to 18x transfer latency reduction over uniform RDMA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。