用错误草稿激发强模型，让数学推理能力大幅提升
Weak-to-Strong Elicitation via Mismatched Wrong Drafts
- 用弱模型生成错误草稿，注入强模型训练上下文
- 在MATH-500上达71.98%准确率，超越现有最优结果
- 无需强化学习或复杂流程，适合高效训练小模型
我们探讨了使用小型弱模型的离策略经验能否激发更强学习者的能力，而标准的在线策略强化学习（如GRPO）无法达到。研究发现，向强模型的GRPO上下文中注入来自更专注领域但数学错误的较小模型的不匹配草稿，能持续优于标准的GRPO方法。实验中以Mathstral-7B为学习者，Qwen2.5-Math-1.5B为草稿模型，基于8.8K道Level 3–5 MATH问题（含500道留出测试集）进行训练。当草稿与当前问题不匹配时，在MATH-500上贪婪通过率提升1.62个百分点（$p=0.0015$），且在所有测试变体中表现最佳。在分布外的AIME 2025/2026数据集上，该方法在样本预算从$ k=1 $到$1024$范围内均显著超越Mathstral-7B原版和草稿模型，尤其在$ k=1024 $时，对AIME 2025提升14.2个百分点，对2026提升9.0个百分点。所有版本均采用相同提示，测试时不注入草稿。该方法仅需单卡训练，无需SFT、奖励模型、合成数据或迭代优化，即可实现71.98%的MATH-500准确率，为当前已知最高成果。
原文摘要 · Abstract (English)
We consider whether off-policy experience from a smaller, weaker model can elicit capability in a stronger learner that on-policy RL fine-tuning (e.g., GRPO) does not reach. We find that injecting mathematically wrong drafts from a smaller but more domain-trained model -- mismatched to the current problem -- into a stronger learner's GRPO context consistently outperforms standard on-policy GRPO on held-out MATH-500 and out-of-distribution AIME 2025/2026. Concretely, we use Mathstral-7B as the learner, Qwen2.5-Math-1.5B as the draft model, 8.8K Level 3--5 MATH problems (with MATH-500 held out), and train with Dr. GRPO. Mismatch is an active ingredient: shuffling drafts to mismatched problems while holding everything else constant yields $+1.62$pp on MATH-500 (greedy pass@1) over the matched-wrong variant ($n=10$ seeds, $p=0.0015$, Welch's $t$). In fact, the mismatched-wrong variant leads all other variants we tested on MATH-500 across both greedy pass@1 and sampling pass@$k$. On out-of-distribution AIME 2025 and 2026, the mismatched-wrong variant uniquely lifts pass@$k$ above both Mathstral-7B (in its native [INST] format) and the Qwen2.5-Math-1.5B draft model at every sample budget from $k=1$ to $k=1024$ across 2 seeds ($+14.2$pp on 2025 and $+9.0$pp on 2026 at pass@1024 over Mathstral-7B), and at pass@1024 also leads no-draft, matched-wrong, and mismatched-correct variants on both years. All variants use the same prompt with no draft injection at test time. The recipe -- trained on a single GPU with no SFT, no reward models, no synthesized data, and no produce-critique-revise inner loop -- reaches 71.98% MATH-500 on Mathstral-7B-v0.1, the highest published result on this model to our knowledge, surpassing the heavier WizardMath pipeline at 70.9% on full MATH (SFT + PPO with process/instruction reward models).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。