通过锚定残差引导,让大模型更好学习复杂推理路径。
Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation

- 用部分特权教师构建本地锚点,分离未来导向的最优提示
- 在长推理任务中提升7.2分,比全特权蒸馏高2.3分
- 适合需要稳定推理过程的复杂任务场景
基于策略的蒸馏(OPD)通过将学生模型与教师在其自身轨迹上的预测分布对齐,在增强大语言模型复杂推理能力方面表现出显著效果。新兴的特权OPD进一步引入包含先验信息(如最优轨迹)的自教师模型,以缓解教师-学生能力差距并提供密集、目标导向的监督。然而,现有方法将特权信息视为整体模仿目标,未能区分局部可实现的推理步骤与未来依赖的最优信号,导致学生被引导匹配超出其局部预测支持的回溯偏差分布,从而倾向于跳过有效中间步骤而采用局部不支持的捷径。为此,本文提出锚定残差基于策略蒸馏(AR-OPD),一种双视角框架,通过部分特权教师建立局部兼容锚点,将最优预见作为受控残差注入,提供目标导向引导。在多种推理任务中,AR-OPD相比全特权OPD提升2.3分,比监督微调(SFT)高7.9分;关键在于,该机制降低21.7%的回溯泄露,缓解后期漂移,在超过768词元的长轨迹任务上实现最高7.2分的优势。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has demonstrated strong empirical gains in enhancing complex reasoning in LLMs by aligning a student model with a teacher's predictive distribution over the student's own trajectories. An emerging variant, Privileged OPD, further strengthens this paradigm by employing a self-teacher model augmented with privileged information, such as oracle traces, to mitigate teacher-student capacity gaps while providing dense, answer-directed supervision. However, current methods treat privileged information as a monolithic imitation target, failing to disentangle locally reachable reasoning steps from future-conditioned oracle signals. Consequently, the student is encouraged to match a hindsight-biased distribution that often falls outside its local predictive support. This reachability mismatch incentivizes the student model to skip valid intermediate reasoning in favor of locally unsupported shortcuts. To resolve this, we introduce Anchored Residual On-Policy Distillation (AR-OPD), a dual-view framework that disentangles privileged supervision. Rather than enforcing strict full-view imitation, AR-OPD establishes a locally compatible anchor using a partially privileged teacher, isolating and injecting oracle foresight as a controlled residual to provide destination-directed guidance. Across diverse reasoning tasks, AR-OPD outperforms full privileged OPD by 2.3 points and SFT by 7.9 points. Crucially, this anchored residual mechanism reduces hindsight leakage by 21.7% and mitigates late-stage drift, yielding up to a 7.2-point advantage on challenging long-horizon trajectories exceeding 768 tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。