arXiv:2605.06638cs.AIcs.CL2026-05被引 1

用可控环境证明:训练方法比模型架构更能决定长程推理能力。

Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

论文配图:Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
图 1 · 摘自论文原文
  • 设计可调难度的逻辑推理框架,分离控制推理深度与逻辑表达力。
  • 计算量随推理深度呈幂律增长,表达力越强,增长越快(指数1.04到2.60)。
  • 更丰富的训练让模型在数学和通用推理上表现更好,且更省算力。

强化学习(RL)已被用于提升大语言模型(LLM)的推理能力,但受限于缺乏可控且可扩展的环境,难以系统研究训练规模与任务难度的关系。观察到的长程推理缺陷曾被认为根植于自回归Transformer架构。为此,我们提出ScaleLogic——一个合成逻辑推理框架,能独立调控两个难度维度:所需证明规划的深度(即推理长度)与底层逻辑的表达力。该框架支持从简单蕴含逻辑(仅“若则”)到包含合取、析取、否定、全称量化等的高阶一阶逻辑。实验表明,训练计算量 $T$ 随推理深度 $D$ 呈幂律关系($T /propto D^γ$,$R^{2} > 0.99$),且幂律指数 $γ$ 随逻辑表达力单调上升,从1.04增至2.60。在下游数学与通用推理基准测试中,更表达力的训练设置带来更大性能提升(最高+10.66分)并实现更优的计算效率转移。结果表明,模型训练内容而非训练量本身,决定其迁移能力。该幂律关系在多种RL方法中均成立,且课程训练显著提升效率。整体说明,长程推理短板并非架构本质问题,可通过改进训练方法与数据解决。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has been applied to improve large language model (LLM) reasoning, yet the systematic study of how training scales with task difficulty has been hampered by the lack of controlled, scalable environments. Observed LLM shortcomings in long-horizon reasoning have raised the prospect that they are fundamental to the autoregressive transformer architecture. To address this, we introduce ScaleLogic, a synthetic logical reasoning framework that offers independent control over two axes of difficulty: the depth of the required proof planning (i.e., the horizon) and the expressiveness of the underlying logic. Our proposed framework supports a wide range of logics: from simple implication-only logic ("if-then") towards more expressive first-order reasoning with conjunction ("and"), disjunction ("or"), negation ("not"), and universal quantification ("for all"). Using this framework, we show that the RL training compute $T$ follows a power law with respect to reasoning depth $D$ ($T \propto D^γ$, $R^{2} > 0.99$), and that the scaling exponent $γ$ increases monotonically with logical expressiveness, from $1.04$ to $2.60$. On downstream mathematics and general reasoning benchmarks, more expressive training settings yield both larger performance gains (up to $+10.66$ points) and more compute-efficient transfer compared to less expressive settings, demonstrating that what a model is trained on, not just how much it is trained, shapes downstream transfer. We further show that the power-law relationship holds across multiple RL methods, and curriculum-based training substantially improves scaling efficiency. More broadly, our results demonstrate that LLM shortcomings in long-horizon reasoning are not fundamental to the underlying architecture, and can be addressed by improved training methodology and data.

强化学习推理能力训练方法逻辑推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。