TETRIS优化批量推测生成，提升大模型推理效率。
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
- 按请求独立筛选最优草稿令牌，减少验证失败
- 相比基线方法，接受率更高，计算资源浪费更少
- 适合资源受限的大模型服务场景
我们提出 TETRIS，一种在多请求批量推测解码中优化整体吞吐量的新方法。与以往针对单个请求或整体请求组进行优化的方法不同，TETRIS 主动为每个请求选择最有可能被接受的草稿令牌，并在并行验证时予以采纳，从而减少被拒绝的令牌数量，降低计算资源浪费。这种高效的资源利用对推理能力有限的服务提供商尤为重要。理论与实证均表明，TETRIS 优于基线推测解码及现有动态草稿令牌选择方法，在大语言模型的批量推理中实现更高效的表现。
原文摘要 · Abstract (English)
We propose TETRIS, a novel method that optimizes the total throughput of batch speculative decoding in multi-request settings. Unlike existing methods that optimize for a single request or a group of requests as a whole, TETRIS actively selects the most promising draft tokens (for every request in a batch) to be accepted when verified in parallel, resulting in fewer rejected tokens and hence less wasted computing resources. Such an effective resource utilization to achieve fast inference in large language models (LLMs) is especially important to service providers with limited inference capacity. Compared to baseline speculative decoding, TETRIS yields a consistently higher acceptance rate and more effective utilization of the limited inference capacity. We show theoretically and empirically that TETRIS outperforms baseline speculative decoding and existing methods that dynamically select draft tokens, leading to a more efficient batch inference in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。