将强化学习中的知识蒸馏重构为概率迁移，提升学生模型对教师偏好路径的精准学习。
Distillation as Probability Transport: Routed On-Policy Distillation
- 将教师-学生差异拆解为概率源与目标，构建显式迁移配对。
- 在4个推理任务上优于传统方法，路由保真度更高、背景干扰更少。
- 适合追求高精度策略学习的强化学习研究者。
在策略蒸馏中，教师知识通过学生生成的轨迹传递，但高效采样导致教师分布被简化为单个标记的标量奖励。该奖励仅指示某标记应增或减概率，却未指定具体分配方式。本文将此过程重新建模为教师引导的概率迁移，提出路由型策略蒸馏（RouteOPD），将局部教师-学生差异分解为学生过量源与教师短缺点，并配对形成显式迁移路径。该方法优化成对对数几率，以有界教师势能获得联合可实现的目标，同时根据教师需求集中度动态调整迁移预算。这一框架使更新方向指向教师偏好的状态，且控制幅度于单一迁移算子内。在四个教师-学生设置及四个数学推理基准上的实验表明，RouteOPD始终优于采样反向KL蒸馏，性能提升伴随更高路由保真度和更低背景泄漏。结果验证了在策略蒸馏中显式建模概率迁移的有效性。
原文摘要 · Abstract (English)
On-policy distillation (OPD) transfers teacher knowledge on student-generated trajectories, but efficient sampled objectives reduce the teacher distribution to scalar credit on individual tokens. Such credit indicates whether a token should gain or lose probability, yet leaves the corresponding redistribution unspecified. We recast OPD as teacher-guided probability transport and propose RouteOPD (Routed On-Policy Distillation), which decomposes local teacher--student disagreement into student-excess sources and teacher-deficit destinations and couples them into explicit transport pairs. RouteOPD optimizes pairwise log-odds toward jointly realizable targets obtained from a bounded teacher potential, while adapting the transport budget to the concentration of teacher demand. This formulation directs updates toward teacher-preferred destinations and controls their magnitude within a single transport operator. Experiments across four teacher--student settings and four mathematical-reasoning benchmarks demonstrate that RouteOPD consistently outperforms sampled reverse-KL OPD, with improvements accompanied by higher routing fidelity and lower background leakage. These results demonstrate the effectiveness of explicitly modeling probability transport in on-policy distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。