让大模型通过迭代推理持续进化，突破训练限制。
Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL
- 用迭代解码替代自回归生成，利用模型总结与生成的不对称性
- 40亿参数模型在测试时用0.5百万词元提升竞赛成绩至近70%
- 适合需要长期推理和持续优化的复杂任务场景
能够超越训练预算持续改进的大语言模型可在测试时适应并解决更难问题，这种能力称为外推。然而，标准强化学习受限于固定的问题分布和训练预算，在测试时分布变化下难以实现外推。为此，我们提出RC——一种迭代解码算法，取代训练和推理阶段的标准自回归解码。RC利用大模型在响应生成与摘要能力上的不对称性，构建可迭代优化的推理链。经训练的模型能在外推推理时程上持续改进，时间跨度比训练时长出一个数量级以上。实证表明，使用16k词元训练预算训练的4B模型，在测试时使用0.5百万词元，使HMMT 2025的成绩从40%提升至近70%，优于同等规模及许多更大模型。此外，因训练中习得的摘要条件生成能力，使用RC训练的模型更能有效利用已有支架（scaffolds）以进一步提升测试性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) that can continually improve beyond their training budgets are able to solve increasingly difficult problems by adapting at test time, a property we refer to as extrapolation. However, standard reinforcement learning (RL) operates over fixed problem distributions and training budgets, which limits extrapolation amidst distribution shift at test time. To address this, we introduce RC, an iterative decoding algorithm that replaces standard autoregressive decoding during both training and inference. RC exploits an asymmetry between the response generation and summarization capabilities of LLMs to construct reasoning chains that consistently improve across iterations. Models trained to use RC can extrapolate and continually improve over reasoning horizons more than an order of magnitude longer than those seen during training. Empirically, training a 4B model with RC using a 16k-token training budget improves performance on HMMT 2025 from 40% to nearly 70% with 0.5m tokens at test time, outperforming both comparably sized models and many larger reasoning LLMs. Finally, we also show that models trained with RC can more effectively leverage existing scaffolds to further scale test-time performance, due to the improved summary-conditioned generation abilities learned through training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。