arXiv:2602.19895cs.LGcs.CL2026-02被引 9

提升大模型推理探索能力,避免陷入固定思维模式。

DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

  • 分尺度设计多样性正则,兼顾全局路径多样性和局部生成随机性。
  • 在多个推理基准上准确率和通过率显著提升,验证深度探索有效性。
  • 适合研究强化学习推理、提升模型泛化能力的学者与工程师。

基于验证器的强化学习(RLVR)是提升大语言模型推理能力的核心范式,但现有方法常受限于探索不足。策略易坍缩至少数推理模式并过早终止深度探索,而传统熵正则仅引入局部随机性,无法激发有意义的路径级多样性,导致群体优化中学习信号弱且不稳定。本文提出双尺度多样性正则化框架DSDR,将推理中的多样性分解为全局与耦合两部分:全局层面促进正确推理轨迹间的多样性以探索不同解法模式;局部层面在正确轨迹上施加长度不变的词元级熵正则,防止每种模式内部的熵坍缩,同时保持正确性。两级通过全局到局部分配机制耦合,强调对更独特正确轨迹的局部正则化。理论证明DSDR在有限正则下保持最优正确性,维持群体优化中有效学习信号,并给出合理的全局到局部耦合规则。在多个推理基准上的实验显示,其在准确率和pass@k上均有持续提升,凸显双尺度多样性对深度探索的关键作用。代码已公开于https://github.com/SUSTechBruce/DSDR。

原文摘要 · Abstract (English)

Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.

强化学习推理增强多样性正则大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。