提出CANON方法,让大模型推理更高效且不依赖预设偏好。
Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
- 根据目标指标分组响应,动态判断正负向影响
- 在数学与逻辑任务上超越现有方法,提升性能与效率
- 无需人工设定优劣方向,适合复杂推理场景
基于可验证奖励的强化学习(RLVR)在大型语言模型(LLMs)中显著提升了数学推理等有明确正确性标准任务的推理能力。已有研究发现熵或响应长度等训练指标与推理行为相关,但以往方法常通过手工设计的惩罚或偏好(如‘越高越好’)进行奖励或优势调整,易因超参数不当导致偏差过大甚至失败。为此,本文提出条件优势估计方法CANON,不预设目标指标的方向性,而是将采样响应按目标指标高低分为两组,通过组间比较确定该指标趋势对表现的影响方向,并在同组内识别更优响应。实验表明,基于熵的CANON在三类大模型上均优于先前方法,在数学推理和高复杂度逻辑任务中表现一致更优;应用于响应长度时,进一步提升生成效率,在性能-成本权衡中实现更优的帕累托前沿。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) for large language models (LLMs) has achieved remarkable progress in enhancing LLMs' reasoning capabilities on tasks with clear correctness criteria, such as mathematical reasoning tasks. Several training metrics, such as entropy or response length, have been observed to correlate with different reasoning behaviors in reinforcement learning. Prior approaches incorporate such priors through reward or advantage shaping, which often relies on hand-crafted penalties and preferences (e.g., higher-is-better or lower-is-better). However, without careful hyperparameter tuning, these directional priors can be overly biased and may lead to failure. To this end, we introduce Conditional advANtage estimatiON (CANON), amplifying the impact of the target metric without presuming its direction. Specifically, CANON regroups the sampled responses into two groups based on the higher or lower value of a target metric, measures which metric trend contributes to better performance through inter-group comparison, and identifies the better response within the same group. In summary, CANON based on entropy consistently outperforms prior methods across three LLMs on both math reasoning and high-complexity logic tasks. When applied to response length, CANON further improves token efficiency, yielding a more favorable Pareto frontier in the performance-cost trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。