arXiv:2509.23667cs.LG2025-09

先对高召回模型对齐,再压缩模型,才能有效学习稀有优质行为。

Why Alignment Must Precede Distillation: A Minimal Working Explanation

  • 先对高召回参考模型进行对齐,再做知识蒸馏。
  • 低召回参考模型导致稀有优质行为对齐失败,奖励降低30%以上。
  • 适用于大模型对齐与压缩的全流程设计,尤其适合强化学习场景。

为提升效率,偏好对齐常在小型化、知识蒸馏(KD)后的模型上进行。我们指出这一常见做法忽视了对齐参考模型的关键属性——分布召回能力。标准的 KD → 对齐流程会削弱模型对罕见但理想行为的对齐能力,即使在强偏好信号下亦然。我们证明反转流程(即 对齐 → KD)至关重要:必须在高召回参考模型上完成对齐后再进行蒸馏。贡献包括:第一,给出参考模型如何从根源上限制对齐目标的最小工作机制解释;第二,在可控的混合高斯实验中验证,低召回锚定导致性能显著下降;第三,在 SmolLM2 系列大模型对齐任务中发现,经蒸馏后对齐的模型目标奖励与精准度明显更低,而我们的 对齐→蒸馏 流程在目标导向指标上表现更优且方差更小。结果确立参考模型召回率是对齐设计的一阶要素,明确原则:对齐必须先于蒸馏。

原文摘要 · Abstract (English)

For efficiency, preference alignment is often performed on compact, knowledge-distilled (KD) models. We argue this common practice introduces a significant limitation by overlooking a key property of the alignment's reference model: its distributional recall. We show that the standard KD -> Align workflow diminishes the model's capacity to align rare yet desirable behaviors, even under strong preference signals. We instead demonstrate that reversing the pipeline (i.e., Align -> KD) is essential: alignment must first be performed on a high-recall reference before distillation. Our contributions are threefold. First, we provide a minimal working explanation of how the reference model constrains preference alignment objectives at a fundamental level. Second, we validate this theory in a controllable Mixture-of-Gaussians experiment, where low-recall anchoring consistently results in suboptimal model performance. Finally, we demonstrate that the same phenomenon holds in LLM alignment with the SmolLM2 family: models aligned after KD fail to effectively align target behaviors, resulting in substantially lower reward and target precision. In contrast, our proposed Align -> KD pipeline robustly aligns these behaviors, yielding models with superior target-oriented metrics and lower variance. Together, these results establish reference-model recall as a first-order design choice in alignment, offering a clear principle: alignment must precede distillation.

模型对齐知识蒸馏大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。