解决语言模型蒸馏中的特权幻觉问题,提升学生模型推理性能。
DAPD: Dual-Anchored Policy Distillation

- 双锚点蒸馏框架通过两条信息路径对齐师生行为,消除训练与推理阶段的信息不对称。
- 在Qwen3-4B上平均提升2.00分,32B规模下达+2.78,效果随模型增大持续增强。
- 适合追求高推理鲁棒性的大模型后训练场景,尤其关注蒸馏稳定性与泛化能力的团队。
在线策略(自)蒸馏(OPSD)在语言模型后训练中日益普及。它通过赋予教师额外信息来强化其表现,但可能引发‘特权幻觉’:学生学习了依赖特权信息的行为,却无法在推理时复现该信息,却仍表现出仿佛拥有训练时特权信息的假象,最终导致性能下降。本文识别出训练阶段教师与推理阶段学生间的信息不对称是此问题的根本原因。为此,提出双锚点策略蒸馏(DAPD),包含两个层级的锚定机制。双路径锚定(DPA)引入自条件桥梁,沿两条匹配的信息路径对齐参考行为与生成行为,防止特权依赖行为传递至推理阶段的学生。双源锚定(DSA)在参考到生成和生成到参考两个方向应用这些路径,降低对特权参考指导的依赖,同时保持正确性监督。大量实验表明,DAPD显著缓解了特权幻觉,在Qwen3-4B上跨任务平均提升2.00分。尤为突出的是,其增益随模型规模持续扩大,4B时达+2.69,32B时达+2.78。
原文摘要 · Abstract (English)
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。