让模型学习老师隐藏状态的演化轨迹，提升推理能力。
PHF: Privileged Hidden Flow for On-Policy Self-Distillation
- 通过匹配教师模型隐藏状态的流动方向，而非单一位置的值。
- 在相同训练条件下，模型性能提升2.2到1.7点。
- 适合需要高效自我蒸馏的大型语言模型研究者。
在策略自蒸馏（OPSD）中，学生模型通过自身策略采样的轨迹进行训练，目标是匹配拥有已验证解的特权教师模型。现有方法仅监督输出分布，导致特权信息通过逐标记差异间接影响训练，未能直接指导内部计算过程。本文提出特权隐藏流（PHF），额外蒸馏教师模型在相同轨迹上的隐藏状态演化过程。与强制对齐同位置隐藏向量不同，PHF对齐的是选定生成位置间的转移方向和轨迹几何结构，并引入跨层关系项，无需逐点模仿隐藏状态。在相同的100步训练周期下，基于Qwen3-1.7B、4B和8B模型，相比复现的基线，平均@12指标分别提升约+2.2、+1.5和+1.7点。该传输目标对共享轨迹偏移完全不变，局部几何项也对转移方向的正交变换保持不变。消融实验表明，所提方案在结构设计、单通道损失与层选择上均优于基线，证明其作为轻量级隐状态流扩展的有效性。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) trains a reasoning model on rollouts sampled from its own policy by matching a privileged teacher that also sees verified reference solutions. Existing OPSD objectives supervise only the output distribution, so privileged context affects training through a token-level divergence without directly supervising the internal computation that produced that distribution. We propose Privileged Hidden Flow (PHF), which additionally distills how a privileged teacher's hidden states move along the same rollout. Rather than forcing each student hidden vector to match the teacher vector at the same token position, PHF aligns token-to-token transition directions and trajectory geometry over selected generated positions. The all-layer recipe also includes an adjacent-layer relation computed from these same transitions, without pointwise hidden-state imitation. Under the same 100-step training schedule, PHF improves the Average@12 aggregate over our reproduced OPSD baseline on Qwen3-1.7B, 4B, and 8B, with observed gains of about +2.2, +1.5, and +1.7 points. The transport objective is exactly invariant to shared trajectory offsets; its local geometry term is also invariant to orthogonal transformations of transition directions. Ablations distinguish the fixed PHF recipe from pointwise hidden-state matching, single-channel transition losses, and layer-subset choices, supporting PHF as a compact hidden-flow extension to OPSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。