利用错误推理过程训练小模型，显著提升逻辑推理能力。
Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning
- 用正负推理轨迹联合训练，采用强化学习式优化目标
- 仅用13.1万条数据达MATH-500 83.1%准确率
- 适合追求高效推理训练的模型开发者
近期模型压缩研究显示，先进推理模型的数据可有效训练小型学生模型。然而，标准方法通常丢弃错误推理路径——这些宝贵但未被充分利用的数据。本文提出关键问题：如何在离线设置下，有效融合正负推理轨迹以最大化大语言模型的推理性能？我们采用两阶段训练策略：首先在正向轨迹上进行监督微调（SFT），随后引入正负轨迹进行精炼。发现一种类似REINFORCE的目标函数（称为强化蒸馏，REDI）在此场景下优于传统偏好优化方法（如DPO、SimPO）。实证评估表明该方法有效性。特别地，基于13.1万条来自Open-R1开源数据集的轨迹训练的Qwen-REDI-1.5B模型，在MATH-500上取得83.1%得分，媲美使用80万条私有数据训练的DeepSeek-R1-Distill-Qwen-1.5B模型。这一结果展示了利用以往被丢弃的负样本带来的惊人数据效率。
原文摘要 · Abstract (English)
Recent advances in model distillation show that data from advanced reasoning models can effectively train smaller student models. However, standard practices discard incorrect reasoning traces -- valuable, yet underutilized data. This paper addresses the critical question: How can both positive and negative distilled reasoning traces be effectively leveraged to maximize LLM reasoning performance in an offline setting? We employ a two-stage training recipe: first, Supervised Fine-Tuning (SFT) on positive traces, followed by a refinement stage using both positive and negative traces. We find that a simple REINFORCE-style objective, which we term the Reinforcement Distillation (REDI) objective, outperforms established preference optimization methods like DPO and SimPO in this distillation context. Our empirical evaluations demonstrate the effectiveness of this approach. Notably, our Qwen-REDI-1.5B model, trained on just 131k traces from the open Open-R1 dataset, achieves an 83.1% score on MATH-500. Its performance matches that of DeepSeek-R1-Distill-Qwen-1.5B, a model trained on 800k proprietary data. This result showcases the remarkable data efficiency of utilizing previously discarded negative traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。