解决大模型长链推理中自我蒸馏失效问题,让模型真正学会思考。
Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
- 通过分解教师信号,分离出可迁移的推理修正信息
- 用互信息构建目标分布,显著提升长链推理性能
- 适合研究大模型推理能力或改进自我蒸馏方法的研究者
基于策略自蒸馏(OPSD)已成为提升大模型推理能力的有前景范式,其中具备参考答案访问权限的教师模型对学生生成的推理轨迹提供逐标记监督。然而我们发现,OPSD在长链推理(long-CoT)模型上表现持续不佳,仅带来微弱增益且破坏了模型依赖的反思推理能力。通过对教师监督信号的新分解,我们识别出根本原因:教师信号主要受参考答案诱导成分主导,导致模型机械记忆特定参考路径的捷径;而与问题相关、可迁移的推理修正成分则被忽略甚至抑制。为此提出两步解决方案:首先构建仅依赖参考答案的教师模型(相同模型仅条件于参考答案而不含问题),以隔离非可迁移成分;其次通过点互信息(PMI)将残差信号转化为可直接蒸馏的目标分布,过滤掉参考诱导的捷径。在四个长链推理模型和两个数据集上的实验表明,该方法持续优于基线模型和标准OPSD,同时保持模型自然的知识探究行为。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories. However, we find that OPSD consistently fails on long chain-of-thought (long-CoT) reasoning models, yielding at best marginal gains while destabilizing the reflective reasoning capability these models depend on. Through a novel decomposition of the teacher's supervision signal, we identify the root cause: the teacher's supervision is dominated by a reference-induced component that drives rote memorization of reference-specific shortcuts, while the question-conditioned, inference-transferable component is ignored or actively opposed. Based on this diagnosis, we propose a two-step solution. First, we construct a reference-only teacher (the same model conditioned on the reference without the question) to isolate the non-transferable component of the supervision signal; the residual after subtracting this component captures the question-conditioned, inference-transferable correction. Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution that the student can directly distill from, filtering out the reference-induced shortcut. Experiments on four long-CoT models across two datasets demonstrate consistent improvements over both the base model and standard OPSD, while preserving the models' natural epistemic behavior throughout training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。