用风险敏感目标提升大模型推理能力,避免过拟合高概率路径。
RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
- 采用混合风险价值目标,关注难样本的奖励分布
- 在数学与代码任务上超越GRPO,Pass@1和Pass@k均显著提升
- 适合需要稳定推理训练的LLM后训练场景
基于可验证奖励的强化学习已成为大语言模型后训练的核心范式;然而,现有基于均值的方法(如GRPO)存在熵崩溃和推理提升有限的问题。我们认为根源在于过度关注高概率输出序列,忽视了罕见但信息量大的推理路径。为此,提出风险感知策略优化(RiskPO),以严谨的风险度量替代传统均值目标。具体地,引入混合价值-at-风险目标,对奖励分布多个区域加权注意力,增强困难实例的梯度信号,防止过自信收敛。同时设计问题捆绑机制,将多个问题打包处理,丰富反馈信号,实现更稳定、更具信息量的训练动态。理论上证明风险规避更新可缓解熵崩溃并促进探索。数值实验表明,RiskPO在数学推理、多模态推理和代码生成任务中持续显著优于GRPO及其变体,在Pass@1和Pass@k指标上均取得突破。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable reward has recently emerged as a central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, such as Group Relative Policy Optimization (GRPO), suffer from entropy collapse and limited reasoning gains. We argue that these issues stem from overemphasizing high-probability output sequences while neglecting rare but informative reasoning paths. To address these challenges, we propose Risk-based Policy Optimization (RiskPO), which substitutes classical mean-based objectives with principled risk measures. Specifically, we introduce a Mixed Value-at-Risk objective that integrates weighted attention over multiple regions of the reward distribution, thereby amplifying gradient signals on challenging instances and preventing overconfident convergence. We further design a bundling scheme that aggregates multiple questions into bundles, thus enriching the feedback signal and yielding more stable and informative training dynamics. Theoretically, we prove that the risk-averse update alleviates entropy collapse and promotes exploration. Numerically, RiskPO achieves consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing GRPO and its variants on both Pass@1 and Pass@k metrics. Our results demonstrate that risk-based optimization provides a rigorous and effective paradigm for enhancing LLM reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。