通过自适应难度树搜索，提升大模型推理覆盖范围。
Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
- 设计难度自适应的树结构采样策略，动态调整探索深度。
- 在数学推理任务中，pass@k提升显著，最高达18.7%。
- 适合追求推理多样性与泛化能力的研究者使用。
强化学习结合可验证奖励（RLVR）是近期大型推理模型成功的关键。然而，尽管显著提升了单样本准确率，传统方法常因训练中探索不足，难以扩展模型内在推理覆盖范围（pass@k）。本文分析提出三大设计原则：(1) 难度自适应的采样不仅提升效率，更关键于扩大pass@k；(2) 树状采样优于并行采样，更易发现正确答案；(3) 基于句熵引导的分叉机制可缓解词级分支的局部化问题，最大化语义多样性。基于此，我们提出DATPO（难度自适应句熵引导树结构策略优化）方法，融合难度自适应树搜索与兄弟节点多样性优势项，显式促进语义多样性以扩展训练期间的推理覆盖。在数学推理基准测试中，DATPO在pass@k指标上显著优于基线，尤其在测试时缩放性能上表现优异，最大提升达18.7%。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。