华为云矩阵384实现超大规模语言模型高效推理。
Serving Large Language Models on Huawei CloudMatrix384
- 采用全互联高速总线架构，支持动态资源池化与高效通信。
- 单NPU预填充吞吐达6,688 tokens/s，解码吞吐1,943 tokens/s。
- 适合需要低延迟高并发的工业级大模型服务场景。
大语言模型（LLM）的快速发展，受参数规模扩大、混合专家（MoE）架构普及和上下文长度增加驱动，对AI基础设施提出前所未有的挑战。传统AI集群在计算强度、内存带宽、芯片间通信和延迟方面面临瓶颈，叠加负载波动和严格的服务等级目标。解决这些问题需软硬件深度融合重构。本文提出华为云矩阵（CloudMatrix）新一代数据中心架构，落地于生产级云矩阵384超节点，集成384个Ascend 910 NPU和192个Kunpeng CPU，通过超高带宽统一总线（UB）网络互联，实现全对全通信与资源动态池化。该设计优化了通信密集型操作性能，如大规模MoE专家并行与分布式键值缓存访问。为充分释放云矩阵384潜力，我们提出CloudMatrix-Infer，包含三项核心创新：点对点服务架构，独立扩展预填充、解码与缓存；支持EP320的大规模专家并行策略，基于高效的UB令牌调度；以及硬件感知优化，包括专用算子、基于微批的流水线和INT8量化。基于DeepSeek-R1模型评估显示，CloudMatrix-Infer达到业界领先效率：单NPU预填充吞吐6,688 tokens/s，解码吞吐1,943 tokens/s（TPOT<50 ms）。在严苛15毫秒延迟约束下仍保持538 tokens/s每NPU吞吐，且INT8量化在各基准测试中维持模型精度。
原文摘要 · Abstract (English)
The rapid evolution of large language models (LLMs), driven by growing parameter scales, adoption of mixture-of-experts (MoE) architectures, and expanding context lengths, imposes unprecedented demands on AI infrastructure. Traditional AI clusters face limitations in compute intensity, memory bandwidth, inter-chip communication, and latency, compounded by variable workloads and strict service-level objectives. Addressing these issues requires fundamentally redesigned hardware-software integration. This paper introduces Huawei CloudMatrix, a next-generation AI datacenter architecture, realized in the production-grade CloudMatrix384 supernode. It integrates 384 Ascend 910 NPUs and 192 Kunpeng CPUs interconnected via an ultra-high-bandwidth Unified Bus (UB) network, enabling direct all-to-all communication and dynamic pooling of resources. These features optimize performance for communication-intensive operations, such as large-scale MoE expert parallelism and distributed key-value cache access. To fully leverage CloudMatrix384, we propose CloudMatrix-Infer, an advanced LLM serving solution incorporating three core innovations: a peer-to-peer serving architecture that independently scales prefill, decode, and caching; a large-scale expert parallelism strategy supporting EP320 via efficient UB-based token dispatch; and hardware-aware optimizations including specialized operators, microbatch-based pipelining, and INT8 quantization. Evaluation with the DeepSeek-R1 model shows CloudMatrix-Infer achieves state-of-the-art efficiency: prefill throughput of 6,688 tokens/s per NPU and decode throughput of 1,943 tokens/s per NPU (<50 ms TPOT). It effectively balances throughput and latency, sustaining 538 tokens/s per NPU even under stringent 15 ms latency constraints, while INT8 quantization maintains model accuracy across benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。