用奖励引导猜测解码，让大模型推理更快更准。
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
- 用轻量模型猜答案，奖励模型决定何时调用强模型
- 推理效率提升4.4倍，准确率比并行解码高3.5%
- 适合资源紧张下需要高质量推理的场景
我们提出一种名为奖励引导猜测解码（RSD）的新框架，旨在提升大语言模型（LLM）推理的效率。RSD将轻量级草稿模型与更强大的目标模型协同使用，通过引入可控的奖励偏差，优先生成高奖励输出，区别于现有方法中严格的无偏性要求。RSD采用过程奖励模型评估中间解码步骤，并动态决定是否调用目标模型，以优化计算成本与输出质量之间的权衡。理论证明，基于阈值的混合策略可实现资源利用率与性能的最佳平衡。在包括奥数级别任务在内的多个挑战性推理基准上的广泛测试表明，RSD相比仅使用目标模型的解码方式，最多减少4.4倍的浮点运算量（FLOPs），且平均准确率高于并行解码方法达3.5%。这些结果表明RSD是一种鲁棒且低成本的LLM部署方案。代码已公开于https://github.com/BaohaoLiao/RSD。
原文摘要 · Abstract (English)
We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios. The code is available at https://github.com/BaohaoLiao/RSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。