用强化学习动态优化大模型生成树结构，显著提速且不损失质量。
Speculative Sampling with Reinforcement Learning
- 通过强化学习实时调整生成树的超参数，实现自适应加速。
- 在五个基准上最高提速5.45倍，比当前最优方法快1.12倍。
- 适合追求推理速度、对生成质量要求高的实际应用开发者。
大语言模型的推理延迟仍是实际应用中的关键挑战。当前最先进的推测采样（SpS）方法如EAGLE-3采用基于树的草稿机制并行探索多个候选延续，但其控制树结构的超参数为静态设置，限制了在不同上下文与领域中的灵活性和效率。本文提出基于强化学习的推测采样（Re-SpS），是首个用于草稿树超参数优化的强化学习框架。Re-SpS 实时动态调整草稿树超参数，学习上下文感知策略，在推测激进性与计算开销之间取得平衡以最大化生成速度。它利用目标模型隐藏状态构建高效状态表示，并引入多步动作持续机制以增强上下文建模能力。在五个多样化基准上的评估结果表明，相比SOTA方法EAGLE-3，Re-SpS实现了稳定提升，最高达到5.45×的生成速度加速，相较EAGLE-3最快提升1.12×，且无输出保真度损失。
原文摘要 · Abstract (English)
Inference time latency has remained an open challenge for real world applications of large language models (LLMs). State-of-the-art (SOTA) speculative sampling (SpS) methods for LLMs, like EAGLE-3, use tree-based drafting to explore multiple candidate continuations in parallel. However, the hyperparameters controlling the tree structure are static, which limits flexibility and efficiency across diverse contexts and domains. We introduce Reinforcement learning for Speculative Sampling (Re-SpS), the first reinforcement learning (RL)-based framework for draft tree hyperparameter optimization. Re-SpS dynamically adjusts draft tree hyperparameters in real-time, learning context-aware policies that maximize generation speed by balancing speculative aggression with computational overhead. It leverages efficient state representations from target model hidden states and introduces multi-step action persistence for better context modeling. Evaluation results across five diverse benchmarks demonstrate consistent improvements over the SOTA method EAGLE-3, achieving up to 5.45$\times$ speedup over the backbone LLM and up to 1.12$\times$ speedup compared to EAGLE-3 across five diverse benchmarks, with no loss in output fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。