用智能筛选减少大模型服务配置的试错成本，省下近六成测试时间。
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
- 按决策影响大小动态选测配置，避免无效实验。
- 实测节省22,200 GPU秒，比随机测试少5.4%~21.5%。
- 兼顾吞吐和延迟，防止违反30秒服务承诺。
为大模型服务集群配置张量并行度和副本数时，性能非单调且随负载变化，难以确定最优方案。传统全量探测耗时费力，仅部分配置影响最终决策。本文提出FleetSieve，根据配置对资源耦合、SLO敏感的决策影响选择测量点。该方法联合建模容量与尾部延迟，比较保守与乐观分配策略，当决策差距低于容忍阈值时停止。在31B参数模型固定H100测量网格下，FleetSieve以22,200 GPU-seconds达成最优决策，比均匀随机探测少6.9%；200次随机测试中平均节省5.4%（95%置信区间3.5–7.2%）。对聊天场景节省达21.5%，但代码生成场景未达最低能耗。联合建模可避免选择完成时间p99达46.4秒、超出30秒SLO的配置。16卡分配中错误决策导致最高损失1.93请求/秒与12.4个百分点的最小最大满足率。边界重复与BurstGPT测试验证了负载依赖的尾延迟机制。
原文摘要 · Abstract (English)
Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty, but measures many configurations that do not affect the final resource allocation. We present FleetSieve, which selects measurements according to their expected effect on a resource-coupled, SLO-aware fleet decision. FleetSieve models capacity and tail latency jointly, compares conservative and optimistic allocations, and stops when their remaining decision gap is below a specified tolerance. On a fixed H100 measurement grid for a 31B-parameter open-weight model, FleetSieve reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling in the fixed comparison. Across 200 random reveal orders, its mean saving over random profiling is 5.4% (95% bootstrap CI: 3.5-7.2%). The fixed-comparison saving is 21.5% for Chat, while FleetSieve does not use the fewest GPU-seconds for Code. Joint capacity and tail modeling also avoids selecting a configuration whose 46.4-second completion p99 violates a 30-second SLO. In a 16-GPU allocation, an incorrect sparse-profile decision loses up to 1.93 requests/s and 12.4 percentage points of max-min fulfillment. Boundary repeats and BurstGPT measurements support the observed load-dependent tail-latency mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。