提出稀疏资格迹机制,精准分配强化学习中的奖励,提升大模型推理效率。
Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR
- 用稀疏资格迹仅对高信息量词元分配奖励,避免无效传播
- 在Qwen3-8B上提升2.98%,样本与令牌效率同步提高
- 适合追求高效推理训练的AI研发人员和模型优化者
基于可验证奖励的强化学习(RLVR)已成为提升大语言模型推理能力的关键方法。然而,广泛使用的无评价器算法如组相对策略优化(GRPO)依赖于‘均匀奖励分配’假设,盲目广播轨迹级优势,无法区分关键推理步骤,影响学习效率。为此,我们提出选择性资格迹(S-trace)。基于部分信任区域保持的直觉,先引入P-trace作为高效、无评价器的资格迹方法,再构建S-trace,通过选择性屏蔽低熵词元实现稀疏资格迹,进一步降低方差并实现细粒度奖励分配。理论上,我们将近期的组序列策略优化(GSPO)置于无评价器资格迹框架下,识别其为均匀奖励分配下的特例。实验表明,S-trace不仅优于GRPO:在Qwen3-1.7B上提升0.49%,在Qwen3-4B上提升3.16%,且在扩展至Qwen3-8B时仍保持2.98%的平均pass@16提升,同时具备更高的样本与令牌效率。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models. However, widely used critic-free algorithms such as Group Relative Policy Optimization (GRPO) necessitate a ``uniform credit assignment'' assumption that indiscriminately broadcast trajectory-level advantages, hindering learning efficiency by failing to distinguish critical reasoning steps. To address this limitation, we propose Selective Eligibility Traces (S-trace). Grounded in the intuition of partial trust region preservation, we initially introduce P-trace as a sample-efficient, critic-free eligibility traces method, upon which we build S-trace, implementing a sparse eligibility traces mechanism to further mitigate variance and achieve fine-grained credit assignment by selectively masking low-entropy tokens. Theoretically, we contextualize the recent Group Sequence Policy Optimization (GSPO) method within the critic-free eligibility traces framework, identifying it as a special instance of the eligibility traces method operating under uniform credit assignment. Experiments demonstrate that S-trace not only outperforms GRPO, showing gains of 0.49\% on Qwen3-1.7B and 3.16\% on Qwen3-4B, and maintaining a robust 2.98\% improvement when scaled further to Qwen3-8B in average pass@16, but notably achieves this with simultaneously higher sample and token efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。