arXiv:2603.21177cs.LGcs.AI2026-03被引 4

通过复用高信息量提示,加速GRPO训练并节省算力。

Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts

  • 仅复用提示而非完整轨迹,保持在线策略优化
  • 优先选择正确率约50%的提示,最大化学习信号
  • 适合提示生成耗时大、数据难度高的场景

基于可验证奖励的强化学习(RLVR)在提升大模型推理能力中至关重要,但GRPO训练受制于昂贵的采样过程,且大量提示无效。本文提出Prompt Replay,一种无额外开销的在线数据选择方法,仅复用提示而非完整轨迹,以维持在线策略优化。每步后将中等难度提示存入缓冲区,优先选取正确率接近0.5(半对半错)的提示,以最大化优势信号。训练批次由重用提示与新样本混合构成,通过冷却步数和最大复用次数控制激进程度与过拟合风险。在多个模型家族(Llama-3.2-3B、Qwen3-8B)和数据集(Dolci、Polaris)上,使用六项标准数学基准的平均准确率评估,Prompt Replay显著减少零方差提示,提升平均绝对优势,并实现更快的初期准确率增长。然而,因配置过于激进,最终收敛于基线水平。该方法在采样为瓶颈且数据难度高时最有效。此外,观察到Qwen2.5-Math存在虚假奖励效应,会干扰消融实验,警示其不宜作为GRPO研究唯一测试平台。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) plays a crucial role in expanding the capacities of LLM reasoning, but GRPO-style training is dominated by expensive rollouts and wastes compute on unusable prompts. We propose Prompt Replay, an overhead-free online data selection method for GRPO that reuses prompts only (not trajectories), to preserve on-policy optimization. After each step, we insert prompts with medium difficulty into a buffer, and prioritize prompts closer to a pass rate of 0.5 (half answers correct, half wrong) to maximize the advantage, thus learning signal. Training batches are formed by mixing reused prompts with fresh samples, with cooldown steps and max reuse times controlling aggressiveness vs risk of overfitting. Across multiple model families (Llama-3.2- 3B, Qwen3-8B) and training datasets (Dolci, Polaris), evaluated using average accuracy on six standard math benchmarks, Prompt Replay reduces zero-variance prompts, increases mean absolute advantage and shows faster initial accuracy gains. Yet, it plateaus and converges with the baseline, as too aggressive configuration was used. The method is most efficient when the rollouts are the primary bottleneck and the dataset is difficult for the model. We additionally observe that Qwen2.5-Math can exhibit spurious-reward effects that invalidates ablations, raising a warning signal for using it as a sole testbed for GRPO method research.

强化学习提示复用GRPO高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。