让大模型学会特定身份,且不损害通用能力。
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation

- 用引导采样将特殊上下文注入训练数据,确保目标行为真实存在。
- 仅在特权信息支持的位置更新模型,避免副作用扩散。
- 适合需要个性化能力又不想破坏通用性的场景。
基于策略的蒸馏(OPD)因其在不引入模型分布漂移的情况下提升大模型能力而受到关注,尤其在无需额外教师模型的自蒸馏(OPSD)中,可利用推理时不可见的私有上下文(如角色设定、私密事实或解题过程)提升性能。然而,这种特权信息可能过度改变模型行为,导致推理风格、响应长度或局部词元偏好等非预期变化,使学生模型学习到的是副作用而非可迁移的行为。本文研究罕见身份/标记场景下的该问题,提出证据引导的基于策略蒸馏(EDGE-OPD),其核心为:1)通过引导采样在采样阶段注入特权上下文行为,确保目标行为出现在在线数据中;2)引入证据掩码机制,仅在特权信息支持的词元位置更新学生模型,而非全序列更新。实验表明,传统OPSD及变体RLSD(含验证器与否)均无法学习目标身份,而引入引导采样后可成功实现;掩码区域消融分析显示,人格信号集中在正证据尾部,揭示了高效知识迁移与通用能力保留的规律。
原文摘要 · Abstract (English)
On-Policy Distillation (OPD) has gained wide attraction as an LLM post-training paradigm due to its effectiveness in improving capabilities without introducing model distribution drift, and consequently, regression in general tasks. On-Policy Self-Distillation (OPSD) is an efficient use-case of OPD, which is appealing as it requires only a single model as a student and teacher, and it also has the benefit of providing privileged context that is a absent at inference time (e.g. a persona, a private fact, or a worked solution) to the teacher during the training process. The challenge in this approach is that the privileged information can change model behavior more than intended: it can modify reasoning, degrade general capabilities, and affect performance indicators like response length, style, or local token preferences. Consequently, OPSD may train the student on side effects rather than a desired, transferable behavior. In this paper, we study this problem in a rare-token/identity setting and propose EviDence GuidEd On-Policy Distillation (EDGE-OPD), a modification of OPSD with two distinct characteristics: a) it uses guided rollouts to inject privileged-context behavior to the student at sampling time, so that the rare target behavior is actually present in the on-policy data, and b) it applies an evidence mask: the student is updated only at token positions where the privileged context supports the sampled token, rather than on every token in the rollout. We empirically show that OPSD (and its variant RLSD, with and without a verifier) completely fail to learn a target identity, while the integration of guided rollouts allows them to succeed. Additionally, mask-region ablations show that the persona signal is localized to the positive-evidence tail, allows us to draw valuable insights about efficient knowledge transfer and preservation of general purpose capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。