ReSpec让强化学习训练大模型更快更稳，提速近4.5倍。
ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
- 动态调整推测解码参数，适应不同训练阶段
- 用知识蒸馏更新草稿模型，避免过时和性能下降
- 按回溯奖励加权更新，保障训练收敛与稳定性
通过强化学习（RL）微调大语言模型（LLM）常受限于生成阶段，占训练时间超75%。推测解码（SD）可加速推理生成，但在RL训练中行为尚不明确。我们发现三个关键问题：大规模批处理下速度提升减弱、持续更新策略导致草稿模型滞后、草稿模型引发策略退化。为此提出ReSpec系统，通过三项机制解决：动态调节SD配置、用知识蒸馏演化草稿模型、按回溯奖励加权更新。在Qwen系列模型（3B–14B）上，实现最高4.5倍加速，同时保持奖励收敛与训练稳定，为高效基于RL的LLM适配提供实用方案。
原文摘要 · Abstract (English)
Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75\% of the training time. Speculative decoding (SD) accelerates autoregressive generation in serving systems, but its behavior under RL training remains largely unexplored. We identify three critical gaps that hinder the naive integration of SD into RL systems: diminishing speedups at large batch sizes, drafter staleness under continual actor updates, and drafter-induced policy degradation. To address these gaps, we present ReSpec, a system that adapts SD to RL through three complementary mechanisms: dynamically tuning SD configurations, evolving the drafter via knowledge distillation, and weighting updates by rollout rewards. On Qwen models (3B--14B), ReSpec achieves up to 4.5x speedup while preserving reward convergence and training stability, providing a practical solution for efficient RL-based LLM adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。