让大模型学会从错误推理中纠错,提升数学题解题鲁棒性。
Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?
- 在训练中引入带错的思维链,教会模型识别并修正错误
- 对带错题目准确率从19%提升至24%,清洁题仍保持41%
- 尤其训练推理错误比计算错误更有效,混合训练最佳
链式思维(CoT)提示已成为大语言模型数学推理的核心方法,但模型对早期错误极为脆弱:一次算术失误或无依据推理通常会传播至最终错误答案。我们探究在故意包含错误的推理轨迹上训练,能否使模型学会检测并恢复错误,同时不损害标准问题求解能力。基于MATH-lighteval中的竞赛级问题,我们生成含有一个控制性错误的CoT前缀(计算错误如符号翻转、遗漏项,或推理错误如误用规则、无依据逻辑步骤),并使用GRPO对Qwen3-4B进行微调,采用二元最终答案奖励。混合错误强化学习模型在干净问题上表现与标准RL相当(41% vs 41%),但在预设错误问题上显著优于后者(24% vs 19%)。值得注意的是,仅在干净数据上训练的RL导致鲁棒性下降至19%,低于未微调基线(20%),表明传统训练反而增加对误导性输入的敏感性。不同错误类型中,针对推理错误的训练带来更大鲁棒性提升,混合训练效果最佳。结果表明,在训练中接触错误推理可提升纠错能力而不牺牲准确性,为构建更鲁棒的数学推理大模型提供了新路径。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) prompting has become central to mathematical reasoning in large language models, yet models remain brittle to early errors: a single arithmetic slip or unjustified inference typically propagates uncorrected to an incorrect final answer. We investigate whether training on intentionally flawed reasoning traces can teach models to detect and recover from such errors without degrading standard problem-solving ability. Using competition-level problems from MATH-lighteval, we generate CoT prefixes containing exactly one controlled error, either a calculation error (sign flips, dropped terms) or a reasoning error (misapplied rules, unjustified logical steps), and fine-tune Qwen3-4B with GRPO using a binary final-answer reward. Our Mixed-CoT-RL model matches standard RL on clean problems (41% vs 41%) while substantially outperforming it on problems prefilled with flawed reasoning (24% vs 19%). Notably, clean-only RL fine-tuning degrades robustness below the untuned baseline 19% vs. 20%), indicating that conventional training increases susceptibility to misleading prefills. Among error types, training on reasoning errors yields greater robustness gains than calculation errors alone, with mixed training performing best. These findings demonstrate that exposure to flawed traces during training can improve error-recovery behavior without sacrificing accuracy, suggesting a path toward more robust mathematical reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。