arXiv:2601.05616cs.LG2026-01被引 1

用自生成推理链提升大模型数学推理能力,不依赖强化学习。

Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworks

  • 两阶段训练:先生成带验证的长思维链数据,再用难度感知采样优化
  • 推理链长度扩展4倍,AIME24竞赛题准确率显著提升
  • 仅靠监督微调就能激活模型内在推理能力,适合资源有限场景

近年来,大语言模型在数学推理等复杂任务中展现出巨大潜力。然而,现有研究主要依赖强化学习框架,忽视了监督微调(SFT)方法。本文提出一种两阶段训练框架,通过自生成的长链思维(CoT)数据增强模型的自我纠错能力。第一阶段采用多轮对话策略,引导模型生成包含验证、回溯、子目标分解和逆向推理的CoT数据,并通过预设规则筛选高质量样本用于SFT。第二阶段引入难度感知拒绝采样机制,动态优化数据分布,强化模型应对复杂问题的能力。该方法使推理链长度延长4倍以上,同时保持良好可扩展性,证明SFT能有效激活模型内在推理能力,为复杂任务优化提供高效路径。实验表明,在GSM8K和MATH500等数学基准上性能提升显著,尤其在AIME24这类竞赛级题目上表现突出。代码将开源。

原文摘要 · Abstract (English)

In recent years, large language models (LLMs) have demonstrated significant potential in complex reasoning tasks like mathematical problem-solving. However, existing research predominantly relies on reinforcement learning (RL) frameworks while overlooking supervised fine-tuning (SFT) methods. This paper proposes a new two-stage training framework that enhances models' self-correction capabilities through self-generated long chain-of-thought (CoT) data. During the first stage, a multi-turn dialogue strategy guides the model to generate CoT data incorporating verification, backtracking, subgoal decomposition, and backward reasoning, with predefined rules filtering high-quality samples for supervised fine-tuning. The second stage employs a difficulty-aware rejection sampling mechanism to dynamically optimize data distribution, strengthening the model's ability to handle complex problems. The approach generates reasoning chains extended over 4 times longer while maintaining strong scalability, proving that SFT effectively activates models' intrinsic reasoning capabilities and provides a resource-efficient pathway for complex task optimization. Experimental results demonstrate performance improvements on mathematical benchmarks including GSM8K and MATH500, with the fine-tuned model achieving a substantial improvement on competition-level problems like AIME24. Code will be open-sourced.

数学推理监督微调思维链双阶段训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。