arXiv:2604.16423cs.LGcs.AI2026-04

对比两种防御训练方法,发现它们保护语言模型不学坏的机制完全不同。

Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integrity

  • 通过行为与机制分析,揭示两种防御方法本质不同
  • PPS能抑制已有坏表现,而IP对已坏模型无效
  • PPS主动逆转激活梯度,IP则模糊特征表示

正向预防引导(PPS)和接种提示(IP)等防御训练方法在阻止大语言模型(LLMs)习得不良特质方面表现出意外效果,二者均在训练中引入诱导性内容。以“邪恶性”为案例,我们通过行为与机制对比发现:两者机制迥异。行为上,二者非单纯关联学习;PPS既能防止新习得,又能减弱已有表达,而IP在已微调过表达该特质的模型中无效。机制上,PPS沿其向量轴将激活梯度转向抑制方向,当与特质表达轴对齐时可反转压力,降低激活;而IP的梯度签名更分散,直接余弦相似性分析显示其与PPS显著不同。此外,IP在诱发数据上降低预测损失,而PPS无此效果,表明IP通过“解释掉”训练数据中的特质表达来起作用。结果揭示二者机制差异,也暴露了对IP机制理解的不足。

原文摘要 · Abstract (English)

Defensive training methods such as positive preventative steering (PPS) and inoculation prompting (IP) offer surprising results through seemingly similar processes: both add trait-inducing objects to large language models (LLMs) during training, and both defend the LLM against acquiring the trait. The surprising success of these methods comes with the question: how do they work? Are PPS and IP doing the same thing? We provide behavioral and mechanistic comparisons of these two methods using "evilness" as a case-study trait. Our central finding is that PPS and IP achieve their defensive benefits through distinct mechanisms. Behaviorally, we show that neither PPS nor IP operates through a purely associative mechanism; and PPS can both defend against trait acquisition and actively reduce pre-existing expression, whereas IP is ineffective in models that were previously finetuned to express the trait. This behavioral divergence is reflected mechanistically: PPS shifts the activation gradient towards an attenuating direction along the PPS vector axis. When the PPS vector is aligned with a trait-expressing axis, it can reverse the gradient pressure, reducing rather than increasing activation along that axis. In contrast, IP continues to resist a precise mechanistic account. Direct cosine similarity analyses reveal that IP has a characteristically different gradient signature than PPS, and qualitative analyses reveal IP's gradient to be more diffuse. Furthermore, IP reduces the next-token prediction loss on trait-expressing data where PPS need not, consistent with the notion that IP "explains away" the trait-expression in the training data. Taken together, our analyses reveal distinct mechanisms by which each method operates and highlight open questions about IP's mechanistic picture.

防御训练语言模型机制分析梯度控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。