用代码推理训练模型，减少思考token消耗，提升效率
Z1: Efficient Test-time Scaling with Code
- 用代码解题轨迹训练模型，学会动态调整思考深度
- 仅需30%的思考token，达到R1-Distill-Qwen-7B性能
- 在通用推理任务上表现良好，适合高效推理场景
大型语言模型可通过测试时计算扩展提升复杂问题求解能力，但往往导致上下文过长、推理令牌成本高。本文提出一种高效的测试时扩展方法：在代码相关推理轨迹上训练模型，降低冗余思考令牌消耗并保持性能。首先构建了包含简单与复杂编码问题及其长短解题路径的Z1-Code-Reasoning-107K数据集；其次提出新的偏移思维窗口（Shifted Thinking Window），通过移除上下文分隔标签（如）并限制推理令牌数量，缓解过度思考开销。使用长短轨迹数据训练并配备该机制的Z1-7B模型，能根据问题复杂度自适应调整推理层级，在多种推理任务中实现高效测试时扩展，平均思考令牌仅为R1-Distill-Qwen-7B的30%，性能相当。值得注意的是，仅用代码轨迹微调的Z1-7B在通用推理任务上也表现出色（GPQA Diamond达47.5%）。对高效推理激发的分析为未来研究提供重要启示。
原文摘要 · Abstract (English)
Large Language Models (LLMs) can achieve enhanced complex problem-solving through test-time computing scaling, yet this often entails longer contexts and numerous reasoning token costs. In this paper, we propose an efficient test-time scaling method that trains LLMs on code-related reasoning trajectories, facilitating their reduction of excess thinking tokens while maintaining performance. First, we create Z1-Code-Reasoning-107K, a curated dataset of simple and complex coding problems paired with their short and long solution trajectories. Second, we present a novel Shifted Thinking Window to mitigate overthinking overhead by removing context-delimiting tags (e.g., <think>. . . </think>) and capping reasoning tokens. Trained with long and short trajectory data and equipped with Shifted Thinking Window, our model, Z1-7B, demonstrates the ability to adjust its reasoning level as the complexity of problems and exhibits efficient test-time scaling across different reasoning tasks that matches R1-Distill-Qwen-7B performance with about 30% of its average thinking tokens. Notably, fine-tuned with only code trajectories, Z1-7B demonstrates generalization to broader reasoning tasks (47.5% on GPQA Diamond). Our analysis of efficient reasoning elicitation also provides valuable insights for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。