提出高效算法，在异构云上实现大模型推理的低延迟、低成本调度。
Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds
- 构建两阶段延迟模型，联合优化模型选择与资源分配。
- AGH算法3秒内完成近优解，支持大规模实例求解。
- 适合高并发、动态负载的云服务场景使用。
在云环境中部署大语言模型（LLM）推理需联合优化模型选择、GPU资源配置、并行策略配置和工作负载路由，以满足延迟、精度、内存和预算约束。尽管混合整数线性规划（MILP）可建模该问题，但其计算开销限制了在需求波动下的频繁重优化。现有启发式方法常独立优化各组件，系统约束下易导致不可行。本文提出一种可扩展的SLO约束下LLM推理框架。将问题建模为包含两阶段延迟模型的MILP，捕捉张量并行与流水线并行下的预填充和自回归解码过程。设计两种约束感知启发式算法：贪心法（GH）和自适应贪心法（AGH）。AGH通过多起点构造、局部搜索与GPU合并增强性能。两者通过并行感知过滤、成本排序与自适应并行缩放保持可行性。基于Azure LLM推理日志的实验表明，GH可在1秒内生成可行解，而AGH在3秒内达到近最优解，且能扩展至精确求解器无法收敛的大规模实例。在超出样本压力测试中（延迟最高增加1.5倍，精度下降），AGH通过预留资源平稳退化，显著降低代价与SLO违反率，优于成本最小化MILP方案。在合成与真实Azure工作负载下，AGH均以更低代价维持SLO合规。结果表明，高质量分配在应对需求波动时具备强鲁棒性，并支持快速适应负载变化。
原文摘要 · Abstract (English)
Serving large language model (LLM) inference in cloud environments requires jointly optimizing model selection, GPU provisioning, parallelism configuration, and workload routing under latency, accuracy, memory, and budget constraints. While mixed-integer linear programming (MILP) can model this problem, its computational cost limits frequent re-optimization under demand variability. Existing heuristics often optimize individual components separately and may become infeasible when system-wide constraints are enforced. This paper presents a scalable framework for SLO-constrained LLM inference. We formulate the problem as an MILP with a two-phase delay model capturing both prefill and autoregressive decoding under tensor and pipeline parallelism. To solve it efficiently, we develop two constraint-aware heuristics: a Greedy Heuristic (GH) and an Adaptive Greedy Heuristic (AGH). AGH extends GH through multi-start construction, local search, and GPU consolidation. Both methods maintain feasibility through parallelism-aware filtering, cost-based ranking, and adaptive parallelism scaling. Experiments based on the Azure LLM Inference Trace show that GH generates feasible solutions within one second, while AGH achieves near-optimal performance within three seconds and scales to large instances where exact solvers fail to converge. Under out-of-sample stress with up to 1.5x delay and accuracy inflation, AGH degrades gracefully through provisioned headroom, yielding substantially lower cost and SLO violations than cost-minimal MILP solutions. Across synthetic and real Azure workloads, AGH maintains SLO compliance at significantly lower cost than exact MILP solutions. These results demonstrate that high-quality allocations provide substantial robustness to demand variability while enabling rapid adaptation to workload changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。