用教师模型与基础模型的差异信号，提升强化学习中推理能力的迁移效率。
On-Policy Delta Distillation
- 用教师模型与基线模型的输出差值作为新教学信号
- 在数学、科学、代码推理任务上显著超越传统方法
- 适合希望快速提升大模型推理能力的研究者
在强化学习中，一种替代性的后训练方法——基于策略的蒸馏，通过教师模型提供逐标记级别的监督，缓解了奖励模型带来的约束。尽管该方法已在多种场景中研究与应用，其基本设计仍缺乏深入探索。本文提出一种新的蒸馏奖励——差值信号（delta signal），不再直接模仿教师模型的输出分布，而是计算教师模型与其基础模型在指令微调前的输出差异。该差值捕捉了推理能力微调所带来的变化，为推理能力迁移提供了更直接的信号。通过大量实验证明，差值信号能显著提升基于策略的蒸馏效果，我们将其称为基于策略的差值蒸馏（OPD²）。在数学、科学和代码推理基准测试中的实验表明，OPD² 持续优于传统方法，使推理型大模型仅需短时间后训练即可达到优秀性能。代码将开源于 https://github.com/naver-ai/opd2。
原文摘要 · Abstract (English)
On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。