arXiv:2512.00908cs.LGcs.AI2025-12ACL被引 11

针对推理大模型,提出基于低熵片段的精准奖励调节方法

Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs

  • 聚焦推理过程中的低熵片段,识别稳定结构模式
  • 在6个数学基准上优于强基线,提升准确率与鲁棒性
  • 适合需要高可靠性推理的场景,如数学证明与代码生成

基于可验证奖励的强化学习(RLVR)已成为提升大语言模型推理能力的核心方法。现有研究通过词元熵探讨探索机制,认为高熵词元应获得更强更新。但多数推理轨迹由低熵片段构成,这些片段蕴含稳定且可复用的结构模式。我们通过定性与定量分析发现,正确回答间低熵片段的重叠度与模型准确率高度相关,而错误回答间的重叠则呈现稳定但无益的模式。受此启发,我们提出LESS框架,对低熵片段进行细粒度优势调节:放大仅存在于正确回答中的片段,抑制仅出现在错误回答中的片段,中和两者共有的片段,同时保留原有强化学习算法中的高熵探索机制。在主流GRPO基础上实现的LESS,在三个模型架构与六个数学基准上均持续优于强基线,显著提升性能下限的鲁棒性。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has become a central approach for improving the reasoning ability of large language models. Recent work studies RLVR through token entropy, arguing that high-entropy tokens drive exploration and should receive stronger updates. However, they overlook the fact that most of a reasoning trajectory consists of low-entropy segments that encode stable and reusable structural patterns. Through qualitative and quantitative analyses, we find that the overlap of low-entropy segments across correct responses strongly correlates with model accuracy, while overlaps involving incorrect responses exhibit stable but unproductive patterns. Motivated by these findings, we propose LESS, a correctness-aware reinforcement framework that performs fine-grained advantage modulation over low-entropy segments. LESS amplifies segments unique to correct responses, suppresses those unique to incorrect ones, and neutralizes segments shared by both, while preserving high-entropy exploration in the underlying RL algorithm. Instantiated on top of the popular GRPO, LESS consistently improves accuracy over strong RL baselines across three backbones and six math benchmarks, achieves stronger robustness of the performance floor.

强化学习推理增强奖励设计大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。