通过滑动前缀减少推理内存占用，实现高效长思考测试时扩展
Prefix Sliding for efficient test-time scaling
- 保留指令前缀和最近数千令牌，丢弃中间无用推理过程
- 无需训练即可提速3倍，支持超过十万令牌的长推理链
- 适合需要深度推理但受限于内存的模型部署场景
测试时扩展通过增加推理计算量来提升性能，例如让语言模型在解题时进行更长时间的思考。然而，由于模型通过全注意力机制保持完整推理轨迹，处理复杂任务时内存开销巨大。我们发现，随着推理推进，大多数中间推理标记的重要性显著下降。这引发了保留它们是否值得的疑问。基于此，我们提出前缀滑动（Prefix Sliding）：在推理过程中丢弃不属于前缀或最近数千令牌的中间内容。前缀包含关键指令与工具，最近令牌则代表当前正在处理的推理状态。该方法将内存使用量固定，不受推理长度影响，从而实现高效的长周期测试时扩展。无需训练，前缀滑动可使现有模型提速3倍且保持性能；结合强化学习训练后，可支持超过十万令牌的推理链并取得更好表现。消融实验表明，其优于总结中间结果或传统滑动窗口方法。代码已开源。
原文摘要 · Abstract (English)
Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive. However, we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost. Based on this insight, we propose Prefix Sliding, which discards tokens during reasoning that are not part of the prefix or the window of the last few thousand tokens. The prefix has key instructions and tools available to the model, while the most recent tokens are the current reasoning the model is working on. This caps the total memory requirement regardless of how long the model reasons, allowing for efficient long-horizon test-time scaling. Without training, Prefix Sliding can make existing models 3x faster while maintaining performance. Training with Prefix Sliding using reinforcement learning can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens. Ablations show Prefix Sliding outperforms summarizing intermediate tokens or vanilla sliding window. Our code is at https://github.com/Muennighoff/prefix-sliding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。