arXiv:2601.11590cs.DCcs.AI2026-01被引 2

拆分多模态模型推理阶段,提升高并发下的系统吞吐量。

EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascend

  • 将编码、预填充、解码三阶段分离部署,灵活调度资源。
  • 在高并发下端到端吞吐提升57.37%至69.48%,满足严格延迟要求。
  • 适合需要高效支持多模态大模型推理的云服务与AI平台。

随着大规模多模态模型的广泛应用,跨文本、图像、音频和视频模态的高效推理变得至关重要。然而,现有系统通常采用紧密耦合的单体架构,在同质硬件上集成编码、预填充和解码阶段,忽视了各阶段计算特性的异构性,导致资源利用效率低下、系统吞吐受限。为此,本文提出EPD-Serve,一种面向多模态模型的阶段级解耦推理服务系统。该系统将推理流程分解为独立的编码、预填充和解码阶段,通过动态编排实现逻辑隔离与灵活共置。基于Ascend互连拓扑,引入编码与预填充间的异步特征预取,以及预填充与解码间分层分组的键值缓存传输机制,提升跨节点通信效率。同时,融合多路径调度、实例级负载均衡及多阶段硬件共置与空间复用策略,更好支撑多样化的多模态工作负载。在多模态理解模型上的全面实验表明,在高并发场景下,相较于PD解耦部署,EPD-Serve实现57.37%-69.48%的端到端吞吐提升,且满足严格SLO约束:首次响应时间(TTFT)低于2000毫秒,每输出词延迟(TPOT)低于50毫秒。结果验证了阶段级解耦在优化多模态大模型推理系统中的有效性。

原文摘要 · Abstract (English)

With the widespread adoption of large multimodal models, efficient inference across text, image, audio, and video modalities has become critical. However, existing multimodal inference systems typically employ monolithic architectures that tightly couple the Encode, Prefill, and Decode stages on homogeneous hardware, neglecting the heterogeneous computational characteristics of each stage. This design leads to inefficient resource utilization and limited system throughput. To address these issues, we propose EPD-Serve, a stage-level disaggregated inference serving system for multimodal models. EPD-Serve decouples the inference pipeline into independent Encode, Prefill, and Decode stages, enabling logical isolation and flexible co-located deployment through dynamic orchestration. Leveraging the Ascend interconnect topology, EPD-Serve introduces asynchronous feature prefetching between Encode and Prefill stages and a hierarchical grouped KV cache transmission mechanism between Prefill and Decode stages to improve cross-node communication efficiency. In addition, EPD-Serve incorporates multi-route scheduling, instance-level load balancing, and multi-stage hardware co-location with spatial multiplexing to better support diverse multimodal workloads. Comprehensive experiments on multimodal understanding models demonstrate that, under high-concurrency scenarios, EPD-Serve improves end-to-end throughput by 57.37-69.48% compared to PD-disaggregated deployment, while satisfying strict SLO constraints, including TTFT below 2000 ms and TPOT below 50 ms. These results highlight the effectiveness of stage-level disaggregation for optimizing multimodal large model inference systems.

多模态推理优化系统设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。