高奖励数据反而降低小模型数学推理能力，因与自身风格不匹配。
The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning
- 用强模型优化数据会改变小模型的推理风格，导致适配成本上升。
- 在Qwen2.5、LLaMA-3、DeepSeek上，自生成数据比优化数据表现更好。
- 提出风格对齐修正法，既保留逻辑正确性又维持原始推理轨迹。
从强大推理模型中进行知识蒸馏是提升小语言模型（SLMs）数学推理能力的常用方法，通常假设奖励模型评分更高的样本能提供更有价值的监督信号。我们发现数学推理蒸馏中存在一个反直觉的‘质量-效用悖论’：由更强的Oracle模型精炼或合成的数据虽在奖励模型下显示更高‘质量’，但在Qwen2.5、LLaMA-3和DeepSeek系列模型上均表现差于由小模型自身生成并经拒绝采样筛选的轨迹。分析表明，Oracle精炼过程将逻辑修复与分布漂移耦合，使数据偏离小模型的原生推理分布，从而增加学习者适配成本，甚至超过逻辑改进带来的收益。为验证该机制，我们提出‘风格对齐精炼’方法，在保留小模型原生推理轨迹的同时实现逻辑修复。该方法有效降低适配成本并恢复下游性能。研究提示，有效的数学推理蒸馏应同时优化感知解题质量和学习者-数据兼容性，而非仅依赖奖励模型分数。数据与代码见https://github.com/Dracoqhl/Quality-Utility-Paradox。
原文摘要 · Abstract (English)
Knowledge distillation from powerful reasoning models is widely used to improve Small Language Models (SLMs) on mathematical reasoning, often assuming that traces with higher reward model scores provide more useful supervision. We identify a counterintuitive \textbf{Quality-Utility Paradox} in mathematical reasoning distillation. Data refined or synthesized by a stronger Oracle obtains higher perceived quality according to reward models, yet consistently underperforms traces generated by the SLM itself and selected through rejection sampling across Qwen2.5, LLaMA-3, and DeepSeek families. Our analysis shows that Oracle refinement couples logical repair with distributional drift away from the SLM's native reasoning distribution. This drift increases the learner's adaptation cost and can outweigh the benefit of improved reasoning logic. To test this mechanism, we introduce \textbf{Style-Aligned Refinement}, which preserves the native trajectory of the SLM while retaining logical repair from the Oracle. This intervention lowers adaptation cost and restores downstream utility. These findings suggest that effective mathematical reasoning distillation should jointly optimize perceived solution quality and learner-data compatibility, rather than relying solely on reward-model scores. The datasets and code are available at https://github.com/Dracoqhl/Quality-Utility-Paradox.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。