arXiv:2601.04731cs.AIcs.CL2026-01ACL

利用模型自身不确定性生成奖励,提升大模型推理强化学习的数据效率

Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models

  • 用策略的内在不确定性作为自监督奖励信号,无需额外模型或标注
  • 在六个基准上相较GRPO提升4.58(Pass@1)和6.66(Pass@K)
  • 适合追求高效推理训练的大模型研究者,尤其关注数据利用率

当前无评判器的强化学习方法在正向同质提示(所有回放均为正确)场景下效率极低,因优势估计为零导致回放浪费。本文提出一种简洁而强大的解决方案——矿工(Miner),将策略的内在不确定性转化为自监督奖励信号,无需外部监督、辅助模型或额外推理开销。方法首创两项关键创新:(1) 标记级焦点信用分配机制,动态放大关键不确定标记的梯度,抑制过度自信标记;(2) 自适应优势校准,无缝融合内在与可验证奖励。在Qwen3-4B和Qwen3-8B基础模型上,于六个推理基准测试中表现优于其他四种算法,相较GRPO在Pass@1上最高提升4.58,在Pass@K上提升6.66。与其他探索增强方法对比进一步验证了两项新机制的优越性。结果表明,隐含不确定性的挖掘对推理模型的高效、可扩展强化学习训练既必要又充分。代码已开源:https://github.com/pixas/Miner。

原文摘要 · Abstract (English)

Current critic-free RL methods for large reasoning models suffer from severe inefficiency when training on positive homogeneous prompts (where all rollouts are correct), resulting in waste of rollouts due to zero advantage estimates. We introduce a radically simple yet powerful solution to \uline{M}ine \uline{in}trinsic mast\uline{er}y (Miner), that repurposes the policy's intrinsic uncertainty as a self-supervised reward signal, with no external supervision, auxiliary models, or additional inference cost. Our method pioneers two key innovations: (1) a token-level focal credit assignment mechanism that dynamically amplifies gradients on critical uncertain tokens while suppressing overconfident ones, and (2) adaptive advantage calibration to seamlessly integrate intrinsic and verifiable rewards. Evaluated across six reasoning benchmarks on Qwen3-4B and Qwen3-8B base models, Miner achieves state-of-the-art performance among the other four algorithms, yielding up to \textbf{4.58} absolute gains in Pass@1 and \textbf{6.66} gains in Pass@K compared to GRPO. Comparison with other methods targeted at exploration enhancement further discloses the superiority of the two newly proposed innovations. This demonstrates that latent uncertainty exploitation is both necessary and sufficient for efficient and scalable RL training of reasoning models. Code is available at https://github.com/pixas/Miner.

强化学习推理模型数据效率自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。