提出一种分离式重放机制，显著降低推理模型训练耗时。
Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
- 将重放分为学习价值排序与策略兼容性判断两步决策
- 在多个推理任务上达到或超越现有重放方法性能
- 适合追求高效训练的强化学习研究者和工程团队
基于强化学习的推理模型后训练正面临重复生成新轨迹的瓶颈，尤其在以环境交互为主的智能体场景中，这成为主要的时间开销。重放可通过复用历史轨迹减轻负担，但现有方法常嵌入探索、经验重构或混合策略优化等复杂流程，难以分离重放本身的贡献。本文聚焦核心问题：仅靠合理重放选择能走多远？提出Headroom-Drift Replay，作为GRPO的一种分组级重放控制原语，将重放决策拆分为两个独立步骤：Headroom根据剩余学习价值对存储组进行排序，Drift根据与当前策略的兼容性进行门控。新鲜的在线策略流保持不变，且不引入额外生成或训练模块。在数学推理、多模态推理和Agentic Search基准上，该单一干预方案优于朴素重放，并在平均指标Mean@32上达到或超过更复杂的重放方法；在以环境交互为主导的Agentic Search中，实现相当的生成质量，同时大幅降低实际运行时间。
原文摘要 · Abstract (English)
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。