拆分推理流程可提升大模型多节点部署的吞吐与响应速度
Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
- 通过系统性测试海量配置，探索推理拆分的最优路径
- 预填充负载重且模型大时，拆分效果最显著
- 动态速率匹配和弹性扩容是实现性能最优的关键
随着推理规模扩展至多节点部署，将推理过程拆分为不同阶段的解耦架构，为改善吞吐与交互性的权衡关系提供了可行方案。尽管开源生态快速发展，实际部署仍受限于优化空间复杂与系统协调难度。本文首次对大规模场景下的推理解耦进行系统研究，在多种工作负载与硬件配置下评估数十万种设计组合。结果表明，解耦在预填充密集型流量和大模型场景中效果最佳。关键发现强调了动态速率匹配与弹性伸缩在实现帕累托最优性能中的决定性作用。研究为高效解耦部署提供了可操作的实践指导，助力平衡系统吞吐与响应延迟。
原文摘要 · Abstract (English)
As inference scales to multi-node deployments, disaggregation - splitting inference into distinct phases - offers a promising path to improving the throughput-interactivity Pareto frontier. Despite growing enthusiasm and a surge of open-source efforts, practical deployment of disaggregated serving remains limited due to the complexity of the optimization search space and system-level coordination. In this paper, we present the first systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across diverse workloads and hardware configurations. We find that disaggregation is most effective for prefill-heavy traffic patterns and larger models. Our results highlight the critical role of dynamic rate matching and elastic scaling in achieving Pareto-optimal performance. Our findings offer actionable insights for efficient disaggregated deployments to navigate the trade-off between system throughput and interactivity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。