通过自演化生成推理链,让小模型数学解题能力逼近大模型。
Self-Evolved Preference Optimization for Enhancing Mathematical Reasoning in Small Language Models
- 让小模型自动生成、纠错并多样化推理路径,形成闭环优化。
- 在MATH 500等5个数据集上超越基础模型,部分媲美GPT-4o。
- 适合想提升小模型数学推理能力的研究者与应用开发者。
大型语言模型虽大幅提升推理能力,但在复杂多步数学问题上仍受错误传播、缺乏自修正和适应性差的限制。现有方法依赖静态微调或提示工程,难以泛化,且高质量偏好数据稀缺。我们提出SPHERE——一种自演化数据生成管道,通过迭代生成、纠错和多样化推理链,增强小语言模型(SLMs)的数学推理能力。该流程分三阶段:(i) 自生成,模型自主构建解题步骤;(ii) 自纠正,识别并修正错误;(iii) 多样性引导,通过多条有效推理路径提升鲁棒性。实验在MATH 500、GSM8K、AIME、AMC和奥数数据集上验证,经SPHERE训练的模型显著优于基线,部分任务表现媲美甚至超越GPT-4o。结果表明,自演化机制可缩小小模型与顶尖大模型间的推理差距,使数学人工智能更可靠、高效、可扩展。
原文摘要 · Abstract (English)
Large language models (LLMs) have significantly improved their reasoning capabilities; however, they still struggle with complex multi-step mathematical problem-solving due to error propagation, lack of self-correction, and limited adaptability to diverse reasoning styles. Existing methods rely on static fine-tuning or prompt engineering, which fail to generalize across problem complexities, while the scarcity of high-quality preference data further hinders reliable reasoning. We introduce SPHERE, a self-evolving data generation pipeline that enhances reasoning in small language models (SLMs) by iteratively generating, correcting, and diversifying reasoning chains. SPHERE operates in three stages: (i) Self-Generation, where the model autonomously constructs problem-solving steps; (ii) Self-Correction, enabling it to identify and rectify errors; and (iii) Diversity Induction, improving robustness through multiple valid reasoning trajectories. This self-evolution mechanism strengthens mathematical reasoning and enhances model reliability. Evaluations on MATH 500, GSM8K, AIME, AMC, and Olympiad show that SPHERE-trained models achieve significant gains over their base versions and match/surpass GPT-4o on certain benchmarks. Our findings demonstrate that self-evolving models can close the reasoning gap between SLMs and state-of-the-art LLMs, making mathematical AI more reliable, scalable, and efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。