提出并发感知的推测解码与在线草稿学习,显著加速大模型推理优化训练。
FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
- 根据并发情况动态调整草稿生成策略,提升解码效率。
- 在多个数学推理数据集上实现2.35倍至2.72倍端到端加速。
- 适合需要高效强化学习训练的大模型研究者使用。
组相对策略优化(GRPO)通过强化学习显著提升了大语言模型的推理能力,但其实际部署受限于训练过程过慢,主要源于每查询需自回归生成多条响应,导致生成阶段成为主要性能瓶颈。尽管推测解码有望加速,但在高并发训练下效果有限。为此,我们提出一种并发感知的推测解码框架,根据实时并发水平动态调整草稿与验证策略,最大化生成过程加速。此外,为解决训练中目标模型演进与固定草稿模型间分布漂移导致的性能下降问题,引入在线草稿学习机制,使草稿模型能持续接收目标模型反馈并自适应更新。在多个数学推理数据集和模型上的实验表明,该方法实现端到端加速2.35倍至2.72倍,显著优于基线方法。代码已开源:https://github.com/yedaotian9/GRPO_speculative。
原文摘要 · Abstract (English)
Group relative policy optimization (GRPO) has demonstrated significant potential in improving the reasoning capabilities of large language models (LLMs) via reinforcement learning. However, its practical deployment is impeded by an excessively slow training process, primarily attributed to the computationally intensive autoregressive generation of multiple responses per query, which makes the generation phase the primary performance bottleneck. Although speculative decoding presents a promising direction for acceleration, its direct application in GRPO achieves limited speedup under high-concurrency training conditions. To overcome this limitation, we propose a concurrency-aware speculative decoding framework that dynamically adjusts the drafting and verification strategy according to real-time concurrency levels, thereby maximizing the acceleration of the generation process. Furthermore, to address performance degradation arising from distributional drift between the evolving target model and the fixed draft model during training, we introduce an online draft learning mechanism that enables the draft model to continuously adapt using feedback signals from the target model. Experimental results across multiple mathematical reasoning datasets and models demonstrate that the proposed method achieves end-to-end speedups of 2.35x to 2.72x, significantly surpassing baseline approaches in efficiency. The code is available at https://github.com/yedaotian9/GRPO_speculative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。