ECHO通过稀疏置信度门控，动态优化大并发下的推理速度。
ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
- 将推测解码重构成预算调度问题，用稀疏门控统一管理批处理
- 在高并发下实现5.35倍的加速，相对性能提升超20%
- 适合工业级大模型部署，尤其适用于Qwen3-235B等超大规模模型
推测解码可加速大语言模型的推理，但在生产级服务中性能常受抑制。现有评估忽略高并发场景下的计算瓶颈，验证阶段的计算成为主要负担。静态树导致大量验证浪费，动态树则面临累积误判和内核不兼容问题。为此，我们提出ECHO，一个集成于SGLang的高并发优化框架，将推测执行重构为带预算的调度问题。关键在于使用稀疏置信度门控，将批次视为统一的超树，弹性调节深度与宽度的预算，协同优化全局验证步数减少与每步效率最大化。跨多种模型规模的全面评估显示，尤其是在工业级模型Qwen3-235B上，ECHO在低负载与高负载场景均优于最先进方法，实现最高5.35倍的墙钟时间加速，并带来超过20%的相对速度提升。
原文摘要 · Abstract (English)
Speculative Decoding promises to accelerate the inference of Large Language Models, yet its efficacy often degrades in production-grade serving. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales-particularly the industrial-grade Qwen3-235B-demonstrate that ECHO consistently outperforms SOTA methods in both low-load and high-load scenarios, achieving up to 5.35x walltime speedup and delivering over 20% relative speedup gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。