用大模型指导小模型纠错，让小模型数学推理更强。
SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction
- 用大模型提取思维模板，引导小模型生成更细致的推理过程。
- 通过跨模型偏好优化，教会小模型根据大模型修正痕迹自我纠错。
- 70亿参数模型在数学题上超越多个大模型，适合资源有限场景使用。
大型语言模型（如GPT-4、DeepSeek-R1和ReasonFlux）在各类推理任务中表现优异。然而，小型模型在复杂数学推理上仍存在困难，主要因其无法有效识别并修正推理错误。现有基于反思的方法虽能实现自省与纠错，但在独立检测推理步骤错误方面仍受限。为此，我们提出SuperCorrect，一种两阶段框架：第一阶段，从大教师模型中提取分层的高阶与细节思维模板，指导学生模型生成更精细的推理思考；第二阶段，引入跨模型协作的直接偏好优化（DPO），使学生模型在训练中学习教师模型的修正轨迹，从而有效定位并修复错误思路。该方法打破学生模型思维瓶颈，习得新技能以应对难题。大量实验表明，SuperCorrect-7B在MATH/GSM8K基准上分别超越DeepSeekMath-7B 7.8%/5.3%，Qwen2.5-Math-7B 15.1%/6.3%，成为所有70亿参数模型中的最新最佳性能。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) like GPT-4, DeepSeek-R1, and ReasonFlux have shown significant improvements in various reasoning tasks. However, smaller LLMs still struggle with complex mathematical reasoning because they fail to effectively identify and correct reasoning errors. Recent reflection-based methods aim to address these issues by enabling self-reflection and self-correction, but they still face challenges in independently detecting errors in their reasoning steps. To overcome these limitations, we propose SuperCorrect, a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model. In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts. In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model by following the teacher's correction traces during training. This cross-model DPO approach teaches the student model to effectively locate and resolve erroneous thoughts with error-driven insights from the teacher model, breaking the bottleneck of its thoughts and acquiring new skills and knowledge to tackle challenging problems. Extensive experiments consistently demonstrate our superiority over previous methods. Notably, our SuperCorrect-7B model significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA performance among all 7B models. Code: https://github.com/YangLing0818/SuperCorrect-llm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。