HeteroScale协调缩放，提升异构解耦大模型推理的资源效率。
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
- 通过拓扑感知调度与统一指标驱动，协同调控预填充和解码阶段
- 生产环境部署后平均显卡利用率提升26.6个百分点，日省数万小时GPU算力
- 适合大规模异构分布式LLM服务，尤其关注资源利用率的工程团队
大规模语言模型（LLM）推理是高度依赖GPU的任务，传统自动扩缩机制在现代预填充-解码（P/D）解耦架构下表现不佳。这种架构虽强大，却带来异构硬件利用低、网络瓶颈及预填充与解码阶段严重失衡等挑战。我们提出HeteroScale——一种协调式自动扩缩框架，融合拓扑感知调度器与基于首次大规模生产级信号实证研究的新型指标驱动策略。通过单一稳健指标联合扩缩预填充与解码资源池，保持架构平衡并实现高效自适应管理。该方案已在包含数万张GPU的生产环境中部署，显著提升平均GPU利用率26.6个百分点，每日节省数十万GPU小时，同时满足严格的服务水平目标。
原文摘要 · Abstract (English)
Serving Large Language Models (LLMs) is a GPU-intensive task where traditional autoscalers fall short, particularly for modern Prefill-Decode (P/D) disaggregated architectures. This architectural shift, while powerful, introduces significant operational challenges, including inefficient use of heterogeneous hardware, network bottlenecks, and critical imbalances between prefill and decode stages. We introduce HeteroScale, a coordinated autoscaling framework that addresses the core challenges of P/D disaggregated serving. HeteroScale combines a topology-aware scheduler that adapts to heterogeneous hardware and network constraints with a novel metric-driven policy derived from the first large-scale empirical study of autoscaling signals in production. By leveraging a single, robust metric to jointly scale prefill and decode pools, HeteroScale maintains architectural balance while ensuring efficient, adaptive resource management. Deployed in a massive production environment on tens of thousands of GPUs, HeteroScale has proven its effectiveness, increasing average GPU utilization by a significant 26.6 percentage points and saving hundreds of thousands of GPU-hours daily, all while upholding stringent service level objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。