动态调控大模型推理分支并行，提升吞吐量同时保证服务质量。
Regulating Branch Parallelism in LLM Serving
- 每步决策是否开启额外解码分支，基于当前资源余量动态调整。
- 在Qwen3-32B上实现1.77倍吞吐提升，且95%以上服务等级承诺达标。
- 适合高并发大模型服务场景，尤其对响应延迟敏感的应用。
近期方法在大模型输出中引入请求内并行性，使独立分支可并发解码。现有服务系统要么全盘接纳分支（贪婪策略），要么设置固定上限（保守策略）。我们发现二者均不可靠：贪婪策略会放大共享解码阶段的延迟，导致批处理请求在串行阶段性能下降；保守策略则放弃本应由分支并行带来的吞吐增益。我们定义因接纳分支而产生的额外延迟为‘分支外部性’，并指出安全并行宽度取决于批处理构成、上下文长度及累积余量，这些因素随工作负载持续变化。为此提出TAPER，一种每步准入控制器，将额外分支视为机会性任务，仅当预测的分支外部性在当前批次余量预算之内时才允许进入。由于分支级调度解耦了计算与内存：各分支共享请求前缀的键值缓存（KV cache），因此扩宽或收缩并行度无需内存回收。在Qwen3-32B上，TAPER相比IRP-Off提升吞吐1.77倍，相比IRP-Eager提升1.48倍，同时保持超过95%的服务等级承诺（SLO）达成率。
原文摘要 · Abstract (English)
Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the branch externality and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce TAPER, a per-step admission controller that treats extra branches as opportunistic work, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, TAPER improves goodput by $1.77\times$ over IRP-Off and by $1.48\times$ over IRP-Eager, while maintaining over $95\%$ SLO attainment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。