用风险敏感强化学习提升大模型推理多样性。
Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models
- 设计风险追逐目标，平衡平均与最大奖励，促进深度探索。
- 在六项数学推理任务上，pass@k显著提升，pass@1保持稳定。
- 仅需少量代码修改，适配多种大模型和任务。
基于可验证奖励的强化学习（RLVR）在提升大语言模型（LLM）复杂推理能力方面表现优异。然而，现有方法存在探索困境：预训练模型初始策略尖锐集中，使标准强化学习算法局限于少数解法，虽提升单解准确率（pass@1），但抑制了解法多样性及多解性能（pass@k），导致RLVR常仅复现已有能力而非发现新推理策略。为此，本文提出风险敏感强化学习框架，采用风险追逐目标，在均值与最大奖励间插值，实现新算法风险敏感GRPO（RS-GRPO），通过放大对困难提示的学习，驱动更深入探索。该方法实现简单，仅需少量代码修改。在六项数学推理基准上，使用五种不同LLM测试，RS-GRPO始终提升pass@k表现，同时维持或增强pass@1准确率。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing Large Language Models (LLMs) on complex reasoning tasks. However, existing methods suffer from an exploration dilemma: the sharply peaked initial policies of pre-trained LLMs confine standard RL algorithms to a narrow set of solutions, boosting single-solution accuracy (pass@1) but suppressing solution diversity and multi-solution performance (pass@k). As a result, RLVR often distills existing capabilities rather than discovering new reasoning strategies. To overcome this, we introduce a Risk-Sensitive Reinforcement Learning framework. Our approach employs a risk-seeking objective that interpolates between mean and maximum rewards, leading to a novel algorithm, Risk-Sensitive GRPO (RS-GRPO), which drives deeper exploration by amplifying learning from challenging prompts. Remarkably, RS-GRPO is simple to implement, requiring only minor code modifications. On six mathematical reasoning benchmarks and with five different LLMs, RS-GRPO consistently improves pass@k performance while maintaining or enhancing pass@1 accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。