用推理树结构优化LLM强化学习的数据调度，提升训练效率和准确率。
Scheduling Your LLM Reinforcement Learning with Reasoning Trees
- 基于推理树结构设计新评分机制，衡量题目学习难度。
- 提出从简单到复杂的课程式调度算法，平均准确率提升3.2%。
- 适合研究大模型强化学习与数据高效训练的学者参考。
使用可验证奖励的强化学习（RLVR）优化大型语言模型（LLMs）可被视为逐步编辑查询的“推理树”过程。该过程涉及探索节点（标记）并动态调整每个节点处的模型策略。结合数据调度，可进一步提升数据效率与准确率。然而，现有RLVR数据调度方法通常依赖路径级指标对查询进行排序，忽视了查询的推理树结构。本文提出一种新指标——推理得分（r-score），根据推理树结构衡量查询的学习难度。基于r-score，我们提出推理树调度（Re-Schedule）算法，构建从结构简单（高r-score）到复杂（低r-score）的课程化调度流程。在六个数学推理基准上的实验表明，Re-Schedule显著提升平均准确率，最高达3.2%。结果验证了该方法的有效性，表明对推理树结构的深入理解为RLVR数据调度提供了更强大且原则性的基础。
原文摘要 · Abstract (English)
Using Reinforcement Learning with Verifiable Rewards (RLVR) to optimize Large Language Models (LLMs) can be conceptualized as progressively editing a query's `Reasoning Tree'. This process involves exploring nodes (tokens) and dynamically modifying the model's policy at each node. When combined with data scheduling, this process yields further gains in data efficiency and accuracy. However, existing RLVR data scheduling methods typically rely on path-based metrics to rank queries, overlooking the reasoning tree structures of these queries. In this paper, we introduce a novel metric, namely Reasoning Score (r-score), which measures the query's learning difficulty based on the structure of its reasoning tree. Based on the r-score, we propose the Reasoning Tree Schedule (Re-Schedule), a scheduling algorithm that constructs a curriculum progressing from structurally simple (high r-score) to complex (low r-score) queries. Experiments on six math-reasoning benchmarks show that Re-Schedule significantly improves average accuracy, achieving gains of up to 3.2%. These strong results validate our approach and demonstrate that a structural understanding of the reasoning tree provides a more powerful and principled foundation for RLVR data scheduling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。