arXiv:2605.07725cs.CLcs.AI2026-05被引 7

小模型推理时用分步强化教师指导,避免错误累积。

SOD: Step-wise On-policy Distillation for Small Language Model Agents

论文配图:SOD: Step-wise On-policy Distillation for Small Language Model Agents
图 1 · 摘自论文原文
  • 按步骤动态调整教师指导强度,降低错误传播风险
  • 在数学、科学和代码任务上提升20.86%性能
  • 0.6B小模型达AIME 2025的26.13%,适合轻量部署

工具集成推理(TIR)难以扩展至小语言模型,因其在长序列工具调用中易出现不稳定且模型容量有限。尽管强化学习方法如组相对策略优化仅提供稀疏的结果级奖励,近期基于教师的在线策略蒸馏(OPD)通过从学生生成轨迹中提供密集的标记级监督而受到关注。然而实验表明,将OPD应用于TIR会引发关键失败模式:错误的工具调用会在后续推理步骤中级联放大,逐步加剧学生与教师之间的差异,导致教师的标记级监督逐渐不可靠。为此,我们提出SOD——一种针对小语言模型代理的分步在线策略蒸馏框架,根据每一步的差异程度自适应地重加权蒸馏强度。SOD可在高差异区域抑制误导性教师信号,同时在对齐状态中保留密集引导。在具有挑战性的数学、科学和代码基准测试中,SOD相比次优基线最高提升20.86%。值得注意的是,我们的0.6B学生模型在AIME 2025上达到26.13%,证明了代理推理能力向轻量级模型的有效迁移。代码已开源:https://github.com/YoungZ365/SOD。

原文摘要 · Abstract (English)

Tool-integrated reasoning (TIR) is difficult to scale to small language models due to instability in long-horizon tool interactions and limited model capacity. While reinforcement learning methods like group relative policy optimization provide only sparse outcome-level rewards. Recently, on-policy distillation (OPD) has gained popularity by supplying dense token-level supervision from a teacher on student-generated trajectories. However, our experiments indicate that applying OPD to TIR leads to a critical failure mode: erroneous tool calls tend to cascade across subsequent reasoning steps, progressively amplifying student-teacher divergence and rendering the teacher's token-level supervision increasingly unreliable. To address this, we propose SOD, a step-wise on-policy distillation framework for small language model agents, which adaptively reweights distillation strength at each step based on step-level divergence. Therefore, SOD can attenuate potentially misleading teacher signals in high-divergence regions while preserving dense guidance in well-aligned states. Experiments on challenging math, science, and code benchmarks show that SOD achieves up to 20.86% improvement over the second-best baseline. Notably, our 0.6B student achieves 26.13% on AIME 2025, demonstrating effective transfer of agentic reasoning to lightweight models. Our code is available at https://github.com/YoungZ365/SOD.

小模型推理增强蒸馏代理系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。