用零数据自动生成高质量推理训练数据,让模型越练越强。
Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolution
- 三智能体协同进化:教师出题、求解器反思、生成器提炼策略。
- 仅用100个初始问题,数学推理平均提升20.2个百分点。
- 生成数据可超越顶尖商用大模型,适合追求低成本推理增强的团队。
大型语言模型在推理任务上的突破依赖于海量高质量数据,通常需人工标注,难以扩展。现有数据合成或蒸馏方法存在质量不稳定、无法动态适应模型演进的问题,导致训练信号不足。为此,我们提出Socratic-Zero,一种完全自主的框架,通过教师、求解器与生成器三智能体的协同进化,从极少量种子问题中生成高质量训练数据。求解器通过成功与失败轨迹的偏好反馈持续优化推理;教师根据求解器弱点动态设计更难问题;生成器则蒸馏教师的出题策略,实现可扩展的高保真课程生成。该闭环系统无需预设任务或标签。实验表明,仅以100个种子问题为起点,Socratic-Solver-8B在七项数学推理基准(AMC23, AIME24-25, Olympiad, MATH-500, Minerva, GSM8K)上相较已有合成方法平均提升20.2个百分点,且在Qwen3与GLM4系列模型上均表现一致优势。更惊人的是,由Socratic-Generator-32B生成的数据使学生模型性能超越多个SOTA商用大模型,包括Qwen3-235B-A22B、DeepSeek-V3.1-671B、GPT-5、Gemini-2.5-Pro、Grok-4和Claude-4.1-Opus。
原文摘要 · Abstract (English)
Recent breakthroughs in large language models (LLMs) on reasoning tasks rely heavily on massive, high-quality datasets-typically human-annotated and thus difficult to scale. While data synthesis or distillation offers a promising alternative, existing methods struggle with inconsistent data quality and an inability to dynamically adapt to the evolving capabilities of the model, leading to suboptimal training signals. To address these limitations, we introduce Socratic-Zero, a fully autonomous framework that generates high-quality training data from minimal seed examples through the co-evolution of three agents: the Teacher, the Solver, and the Generator. The Solver continuously refines its reasoning by learning from preference feedback on both successful and failed trajectories; the Teacher adaptively crafts increasingly challenging questions based on the Solver's weaknesses; and the Generator distills the Teacher's question-design strategy to enable scalable, high-fidelity curriculum generation. This closed-loop system produces a self-improving curriculum-requiring no pre-existing tasks or labels. Remarkably, starting from only 100 seed questions, our Socratic-Solver-8B achieves an average gain of +20.2 percentage points over prior data synthesis methods across seven mathematical reasoning benchmarks (AMC23, AIME24-25, Olympiad, MATH-500, Minerva, and GSM8K), with consistent gains on both Qwen3 and GLM4 series models. Even more surprisingly, synthetic data from Socratic-Generator-32B enables student LLMs to achieve superior performance compared to other state-of-the-art (SOTA) commercial LLMs on these benchmarks, including Qwen3-235B-A22B, DeepSeek-V3.1-671B, GPT-5, Gemini-2.5-Pro, Grok-4, and Claude-4.1-Opus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。