让稀有解法获得更多奖励,提升强化学习解题效果
When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO

- 按解法罕见程度重新分配奖励,避免常见解法垄断得分
- 在高采样预算下,AIME数学题解题准确率提升显著
- 仅增加6%训练时间,适合竞赛数学类问题求解
基于可验证奖励的强化学习(RLVR)通常将每个正确解答视为独立信号。在GRPO中,这种完成级别的一致性导致结构层面的信用偏差:重复出现的正确解法因采样频率高而累积更多正向权重,而罕见解法则获得较少奖励。本文将此现象形式化为多重性引发的结构级信用集中,并提出一种基于聚类稀有度的奖励重分配规则。Cue-GRPO通过确定性策略提示,在无需额外模型推理的情况下构建验证正确轨迹的局部聚类分区来实现该规则。在Qwen2.5-Math-7B和Llama-3.1-8B-Instruct上,Cue-GRPO显著提升了AIME重复采样性能,尤其在高采样预算下增益最大。基于裁判生成分区的信用重分配进一步表明该机制可适配不同分区方式。相较于GRPO,Cue-GRPO仅增加6%的训练开销。结果支持结构级信用重分配作为RLVR的有效设计方向,策略提示提供了竞赛数学任务中低开销的实现方案。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) com- monly optimizes each correct completion as an independent learning signal. In GRPO, this completion-level uniformity creates structure-level skew: recurring correct solution forms accumulate positive coefficient mass in proportion to how often they are sampled, while rare forms receive limited credit. We formalize this behavior as multiplicity-induced structure-level credit concentration and introduce a partition- conditioned rule that redistributes positive advantages accord- ing to cluster rarity. Cue-GRPO instantiates this rule with- out auxiliary-model inference by using deterministic Strategy Cues to construct rollout-local partitions of verified-correct traces. Across Qwen2.5-Math-7B and Llama-3.1-8B-Instruct, Cue-GRPO improves AIME repeated-sampling performance, with the largest gains at high sampling budgets. Credit Re- distribution (CR) under Judge Partitions (JP) further indi- cates that the proposed redistribution mechanism can oper- ate with judge-derived partitions. Cue-GRPO adds only 6% wall-clock training overhead over GRPO. These results sup- port structure-level credit redistribution as a practical design axis for RLVR, with Strategy Cues providing a low-overhead implementation for competition mathematics. Code is avail- able at https://github.com/CzZ12/When-Correct-Solutions- Repeat-Rarity-Aware-Credit-Redistribution-for-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。