让模型学会不依赖外部提示仍能正确推理，且重新引入提示也不会搞砸。
When Context Returns: Toward Robust Internalization in On-Policy Distillation
- 用无提示输出作锚点，通过正向KL散度约束有提示时的输出一致性。
- 在14种设置中，12组减少提示反噬，多数保持或提升无提示性能。
- 适合追求鲁棒性、防止提示干扰的实用型模型部署场景。
近期研究发现，策略内蒸馏可将系统提示或任务提示等特权上下文内化到学生模型中，使推理时无需再提供上下文。然而我们识别出一种反直觉且此前未被研究的现象：重新引入原始特权上下文常导致学生模型性能下降，甚至在原本已正确解决的样本上也如此。我们称此为上下文诱发退化，并认为稳健内化不仅需匹配教师在上下文条件下的行为，还需在特权上下文重新引入时保持稳定，这一理想特性称为上下文不变性。为此，我们提出一种新型视图鲁棒内化风险，并设计轻量级的一致性正则化方法No-Context Anchoring（NCA），利用学生模型无梯度的无上下文输出作为锚点，通过前向KL散度对齐其上下文条件输出。在跨越多个领域和模型家族的14种配置中，NCA在多数情况下提升了上下文条件准确率，且在12/14组中降低了上下文危害，同时保持或提升了无上下文性能，展现出更强的上下文重引入鲁棒性。
原文摘要 · Abstract (English)
Recent work has shown that on-policy distillation can internalize privileged context, such as system prompts or task hints, into a student model so that the context is no longer needed at inference time. However, we identify a counterintuitive and previously unstudied phenomenon: reintroducing the original privileged context to the distilled student often degrades its performance, even on instances it already solves correctly without context. We term this phenomenon context-induced degradation and argue that robust internalization requires not only matching the teacher's context-conditioned behavior, but also remaining stable when the privileged context is reintroduced, a desirable property we call context invariance. To promote this property, we formulate a novel view-robust internalization risk and propose No-Context Anchoring (NCA), a lightweight yet effective consistency regularizer that uses the student's stop-gradient no-context output as an anchor and aligns its context-conditioned output via forward KL divergence. Across 14 configurations spanning diverse domains and model families, NCA improves context-conditioned accuracy in most settings and reduces context harm in 12 out of 14, while preserving or improving no-context performance, demonstrating greater robustness to context reintroduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。