密集自蒸馏未必更优，可能加剧遗忘和崩溃
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
- 用自蒸馏策略优化强化学习，发现密集蒸馏易引发参数与输出漂移
- 在分布外场景下，该方法比传统强化学习遗忘更严重且可能崩溃
- 适合需要稳定教师信号的领域内快速适应，不适用于通用持续学习
持续后训练使基础模型能在保留已有能力的同时学习新知识。近期研究认为，基于策略的学习可缓解遗忘，其中自蒸馏方法尤为吸引人。本文通过自蒸馏策略优化（SDPO）重新审视这一观点。实验表明，当教师信号稳定且对齐时，SDPO能加速领域内专化；但在分布外场景中，其泛化能力差，遗忘更严重甚至发生崩溃。相比之下，如GRPO等基于策略的强化学习方法更具保守性，更利于保持先验能力。进一步分析显示，密集自蒸馏会引发参数空间与响应空间的更大漂移，并通过师生反馈环放大高频格式伪影。这些结果表明，仅靠在线数据不足以支撑持续学习。密集自蒸馏在教师目标稳定、令牌级监督可靠时可加速专化，但不应默认作为持续后训练的稳定器。代码已开源。
原文摘要 · Abstract (English)
Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。