用强化学习优化无限推理，让模型自己决定何时总结和继续。
InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning
- 通过强化学习自动决策总结时机与内容，替代固定规则。
- 在AIME24上准确率提升21%，推理延迟显著降低。
- 适合需要高效长链推理的复杂问题求解场景。
大型推理模型通过扩展推理时的思维链取得优异表现，但该范式存在二次计算开销、上下文长度限制以及中间信息丢失等问题。迭代推理通过周期性总结中间思考缓解上述问题，但现有方法依赖监督学习或固定启发式策略，无法优化总结时机、保留内容及推理续接。本文提出InftyThink+，一种端到端的强化学习框架，通过模型控制的迭代边界与显式总结机制，优化整个迭代推理过程。采用两阶段训练：先监督冷启动，再进行轨迹级强化学习，使模型学会战略性地总结与延续。在DeepSeek-R1-Distill-Qwen-1.5B上的实验表明，InftyThink+在AIME24上准确率提升21%，显著优于传统长思维链强化学习方法，且对分布外基准泛化能力更强。此外，该方法显著降低推理延迟并加速强化学习训练，实现性能与效率双重提升。
原文摘要 · Abstract (English)
Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. Iterative reasoning mitigates these issues by periodically summarizing intermediate thoughts, yet existing methods rely on supervised learning or fixed heuristics and fail to optimize when to summarize, what to preserve, and how to resume reasoning. We propose InftyThink+, an end-to-end reinforcement learning framework that optimizes the entire iterative reasoning trajectory, building on model-controlled iteration boundaries and explicit summarization. InftyThink+ adopts a two-stage training scheme with supervised cold-start followed by trajectory-level reinforcement learning, enabling the model to learn strategic summarization and continuation decisions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B show that InftyThink+ improves accuracy by 21% on AIME24 and outperforms conventional long chain-of-thought reinforcement learning by a clear margin, while also generalizing better to out-of-distribution benchmarks. Moreover, InftyThink+ significantly reduces inference latency and accelerates reinforcement learning training, demonstrating improved reasoning efficiency alongside stronger performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。