让小模型学大模型推理,关键不是模仿每一步,而是按质量动态调整指导强度。
Validity-Calibrated Reasoning Distillation

- 用局部有效性评估替代逐字模仿,动态调节教学信号强度。
- 在数学、编程和指令跟随任务中,性能全面超越现有方法。
- 适合想高效压缩大模型推理能力的研究者和工程师。
推理蒸馏旨在将大型语言模型的多步推理能力迁移到更小、更高效的模型中。尽管近期方法取得了显著进展,但通常依赖于静态的师生层级结构,并将蒸馏视为轨迹模仿。这与推理的内在结构不一致:中间步骤往往局部信息不足——全局正确性约束最终答案,却无法唯一确定每个中间步骤。我们提出有效性校准的推理蒸馏框架,将推理蒸馏视为局部学习信号分配问题,而非路径对齐。不再强制逐标记模仿,而是比较学生与教师在相同前缀下的下一步动作提议,并利用其相对局部有效性来调节蒸馏更新强度。该机制实现了动态、上下文相关的监督,既保留了教师的结构引导,又根据局部推理质量自适应调整更新力度。在数学推理、代码生成和指令遵循等多个基准上,本方法持续优于强基线。结果表明,有效的LLM推理蒸馏并非依赖刚性的轨迹模仿,而是基于有原则的、局部校准的学习信号分配。
原文摘要 · Abstract (English)
Reasoning distillation aims to transfer multi-step reasoning capabilities from large language models to smaller, more efficient ones. While recent methods have shown promising gains, they typically rely on static teacher-student hierarchies and frame distillation as trajectory imitation. This is misaligned with the structure of reasoning, where intermediate steps are often locally under-specified: global correctness constrains the final answer, but does not uniquely determine each intermediate move. We propose validity-calibrated reasoning distillation, a framework that treats reasoning distillation as a problem of local learning-signal allocation rather than path alignment. Instead of enforcing token-level imitation, we compare the student's and teacher's proposed next-step actions under the same prefix and use their relative local validity to modulate the strength of the distillation update. This yields a dynamic, context-dependent supervision mechanism that preserves the teacher's structural guidance while adapting update strength to local reasoning quality. Across mathematical reasoning, code generation, and instruction-following benchmarks, our method consistently outperforms strong distillation baselines. These results indicate that effective LLM reasoning distillation is governed not by rigid trajectory imitation, but by principled, locally calibrated allocation of learning signal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。