arXiv:2601.04992cs.CL2026-01ACL被引 4

用错误推理过程训练模型,能显著提升跨领域泛化能力。

Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization

  • 将错误推理链纳入训练,利用其中有效的中间步骤
  • 在Qwen2.5-7B上实现5.51%的OOD性能提升,MMLU达76.47%
  • 提出GLOW动态加权机制,自动识别并强化有益样本

在链式思维(CoT)演示上进行监督微调(SFT)是增强大语言模型推理能力的常用方法。传统做法仅保留最终答案正确的轨迹(正例),忽略其余(负例)。我们指出,这一范式丢弃了大量监督信号,加剧过拟合,限制了跨领域(OOD)泛化能力。令人惊讶的是,将错误轨迹引入SFT可显著提升OOD泛化效果,因为这些轨迹虽最终答案错误,但往往包含有效中间推理。我们系统分析数据、训练动态与推理行为,发现负例链中存在22种重复模式:它们在训练中减缓损失下降,缓解过拟合;在推理时使策略熵提升35.67%,促进探索。基于此,我们提出基于收益的损失加权(GLOW)方案,根据样本跨轮次进展自适应调整损失权重。实验证明,GLOW高效利用未过滤轨迹,在Qwen2.5-7B上相比仅用正例的SFT获得5.51%的OOD增益,并将MMLU从72.82%提升至76.47%作为强化学习初始化。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) on chain-of-thought (CoT) trajectories demonstrations is a common approach for enabling reasoning in large language models. Standard practices typically only retain trajectories with correct final answers (positives) while ignoring the rest (negatives). We argue that this paradigm discards substantial supervision and exacerbates overfitting, limiting out-of-domain (OOD) generalization. Specifically, we surprisingly find that incorporating negative trajectories into SFT yields substantial OOD generalization gains over positive-only training, as these trajectories often retain valid intermediate reasoning despite incorrect final answers. To understand this effect in depth, we systematically analyze data, training dynamics, and inference behavior, identifying 22 recurring patterns in negative chains that serve a dual role: they moderate loss descent to mitigate overfitting during training and boost policy entropy by 35.67% during inference to facilitate exploration. Motivated by these observations, we further propose Gain-based LOss Weighting (GLOW), an adaptive, sample-aware scheme that exploits such distinctive training dynamics by rescaling per-sample loss based on inter-epoch progress. Empirically, GLOW efficiently leverages unfiltered trajectories, yielding a 5.51% OOD gain over positive-only SFT on Qwen2.5-7B and boosting MMLU from 72.82% to 76.47% as an RL initialization.

推理增强泛化能力训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。