用已有数据训练大模型长程推理,效果比现有方法好得多。
h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
- 把简单问题合成多步复杂链,用结果奖励训练模型
- 在数学题上提升最多达2.06倍,跨任务泛化能力强
- 适合想提升模型长程推理能力的研究者
大语言模型在短程推理任务中表现优异,但随着推理长度增加,性能显著下降。现有方法依赖推理时的结构辅助或昂贵的逐步监督,难以扩展。本文提出一种可扩展的方法,仅使用现有的丰富短程数据,通过合成任意长度的多步依赖链来构建复杂问题,并采用逐步升级难度的课程学习策略,在仅基于结果奖励的情况下进行强化学习训练,使训练可大规模扩展且不饱和。实验表明,该方法在6年级水平数学题(GSM8K)上训练后,对更长、竞赛级基准(GSM-Symbolic、MATH-500、AIME)的准确率提升最高达2.06倍。同时在多种分布外推理任务和长上下文基准上表现出显著迁移能力。更重要的是,即使在高pass@k条件下,其长程推理提升也显著优于基线,证明模型可通过强化学习发现新推理路径。理论上,课程强化学习结合结果奖励在样本复杂度上实现指数级改进,训练信号接近密集监督。
原文摘要 · Abstract (English)
Large language models excel at short-horizon reasoning tasks, but performance drops as reasoning horizon lengths increase. Existing approaches to combat this rely on inference-time scaffolding or costly step-level supervision, neither of which scales easily. In this work, we introduce a scalable method to bootstrap long-horizon reasoning capabilities using only existing, abundant short-horizon data. Our approach synthetically composes simple problems into complex, multi-step dependency chains of arbitrary length. We train models on this data using outcome-only rewards under a curriculum that automatically increases in complexity, allowing RL training to be scaled much further without saturating. Empirically, our method generalizes remarkably well: curriculum training on composed 6th-grade level math problems (GSM8K) boosts accuracy on longer, competition-level benchmarks (GSM-Symbolic, MATH-500, AIME) by up to 2.06x. It also transfers significantly to diverse out-of-distribution ReasoningGym domains and long-context benchmarks, indicating broader generalization. Importantly, our long-horizon improvements are significantly higher than baselines even at high pass@k, showing that models can learn new reasoning paths under RL. Theoretically, we show that curriculum RL with outcome rewards achieves an exponential improvement in sample complexity over full-horizon training, providing training signal comparable to dense supervision. h1 therefore introduces an efficient path towards scaling RL for long-horizon problems using only existing data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。