通过渐进式思维编码,让大模型在有限内存下高效推理。
Training Large Reasoning Models Efficiently via Progressive Thought Encoding
- 将中间推理逐步编码为固定大小向量,避免全程反向传播。
- 在相同内存限制下,数学题准确率提升最高达23.4%。
- 适合需要高效训练复杂推理模型的研究者使用。
大型推理模型(LRMs)在复杂问题上表现优异,但强化学习(RL)训练效率低:需长序列推演获取奖励,自回归解码占据主要时间和内存。滑动窗口缓存虽能控制内存,却破坏长上下文推理,导致性能下降。本文提出渐进式思维编码(Progressive Thought Encoding),一种参数高效的微调方法,使LRMs在固定大小缓存下仍能有效推理。该方法将中间推理逐步编码为固定大小的向量表示,无需对完整缓存序列反向传播,显著降低内存占用,并在推理时保持内存恒定。在Qwen2.5-3B-Instruct、Qwen2.5-7B-Instruct和DeepSeek-R1-Distill-Llama-8B三个模型上,于六个主流数学基准测试中均取得一致提升:相比LoRA微调平均提高19.3%,相比未微调的LRM平均提高29.9%,在AIME2024/2025上最高提升达23.4%。结果表明,该方法不仅提升推理准确率,更使LRM的强化学习训练在真实内存约束下更高效、可扩展。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate reasoning into fixed-size vector representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, on six widely used challenging mathematical benchmarks show consistent gains: our method achieves +19.3% improvement over LoRA-based fine-tuning and +29.9% over LRMs without fine-tuning on average, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。