用强化学习让大模型边想边答,提速省时还更准。
Interleaved Reasoning for Large Language Models via Reinforcement Learning
- 通过强化学习引导模型在回答中穿插思考,实现边推理边生成。
- 相比传统模式,准确率提升12.5%,推理长度减少37%,首次响应时间降低超80%。
- 无需外部工具,在MATH、GPQA等复杂数据集上表现优异,适合高效推理场景。
长链思维(CoT)显著提升大语言模型的推理能力,但冗长的推理过程导致效率低下和首次响应时间(TTFT)增加。本文提出一种仅使用强化学习(RL)的训练范式,引导模型对多跳问题进行思考与回答交错的推理。我们发现模型本身具备交错推理能力,可通过强化学习进一步增强。设计了一种简单有效的奖励机制,激励正确中间步骤,利用交错推理过程中产生的中间信号指导策略模型走向正确推理路径。在五个不同数据集和三种强化学习算法(PPO、GRPO、REINFORCE++)上的实验表明,该方法在不依赖外部工具的情况下,持续优于传统的先思考后回答模式。本方法在提升最终任务准确率的同时,增强了强化学习中的信用分配效率。具体而言,其在Pass@1准确率上提升12.5%,推理总长度减少37%,平均首次响应时间(TTFT)降低超过80%。此外,仅在问答与逻辑推理数据集上训练的模型,对MATH、GPQA和MMLU等复杂推理数据集也展现出强泛化能力。我们还进行了深入分析,揭示了条件奖励建模的若干关键洞见。
原文摘要 · Abstract (English)
Long chain-of-thought (CoT) significantly enhances the reasoning capabilities of large language models (LLMs). However, extensive reasoning traces lead to inefficiencies and increased time-to-first-token (TTFT). We propose a training paradigm that uses only reinforcement learning (RL) to guide reasoning LLMs to interleave thinking and answering for multi-hop questions. We observe that models inherently possess the ability to perform interleaved reasoning, which can be further enhanced through RL. We introduce a simple yet effective reward scheme to incentivize correct intermediate steps, guiding the policy model toward correct reasoning paths by leveraging intermediate signals generated during interleaved reasoning. Extensive experiments across five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++) demonstrate consistent improvements over traditional think-answer reasoning, without requiring external tools. Our method improves final task accuracy and overall efficiency by enabling more effective credit assignment during RL. Specifically, our approach achieves a 12.5% improvement in Pass@1 accuracy, while reducing overall reasoning length by 37% and TTFT by over 80% on average. Furthermore, our method, trained solely on question answering and logical reasoning datasets, exhibits strong generalization to complex reasoning datasets such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to reveal several valuable insights into conditional reward modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。