用自生成推理数据迭代优化模型，不扩规模也能提升推理能力
Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models
- 通过自生成推理轨迹循环训练，结合真实标签剪枝保证数据质量
- Gemma2-2B在GSM8K上准确率从41.9%提升至57.6%，LLaMA-3.1-70B达91%超越GPT-4o
- 适合追求高效推理增强、不想盲目扩模型的AI研发团队
大型语言模型在编程与数学推理任务中表现强劲，但受限于高质量训练数据。合成数据可用于提升微调效果，但其成效受模型规模、数据量、剪枝策略及微调轮次影响。本文探索这些因素，提出可扩展的Think, Prune, Train框架：模型在自身推理轨迹上迭代微调，并通过真实标签剪枝确保训练数据质量。该方法显著提升性能：在GSM8K上，Gemma2-2B Pass@1达57.6%（原为41.9%），Gemma2-9B达82%（媲美LLaMA-3.1-70B），LLaMA-3.1-70B更达91%，超越GPT-4o，验证了自生成推理与系统化数据选择的有效性。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong capabilities in programming and mathematical reasoning tasks, but are constrained by limited high-quality training data. Synthetic data can be leveraged to enhance fine-tuning outcomes, but several factors influence this process, including model size, synthetic data volume, pruning strategy, and number of fine-tuning rounds. We explore these axes and investigate which conditions enable model self-improvement. We introduce the Think, Prune, Train process, a scalable framework that iteratively fine-tunes models on their own reasoning traces, using ground-truth pruning to ensure high-quality training data. This approach yields improved performance: on GSM8K, Gemma2-2B achieves a Pass@1 of 57.6% (from 41.9%), Gemma2-9B reaches 82%, matching LLaMA-3.1-70B, and LLaMA-3.1-70B attains 91%, even surpassing GPT-4o, demonstrating the effectiveness of self-generated reasoning and systematic data selection for improving LLM capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。