arXiv:2510.04340cs.CLcs.AI2025-10被引 31

通过训练时诱导不良特质,可抑制测试时的表达,实现精准控制。

Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time

  • 训练时添加诱导指令,让模型学习不良特质
  • 测试时移除指令,不良特质表达大幅降低
  • 适用于防后门、抗隐性传播,适合安全可控场景

语言模型微调常伴随习得不良特质。为此,我们提出接种提示:在微调数据前添加简短系统指令,刻意诱导不良特质。测试时移除该指令,接种模型的特质表达显著低于未修改数据训练的模型。接种具有选择性:在助手始终用西班牙语且全大写的模拟场景中,恰当的接种(如“你总是用西班牙语说话”)使模型学会大写输出,但仍用英语回应。实验证明,该方法在减少任务微调引发的涌现错位(EM)、防御后门攻击及缓解隐性学习传播方面均有效。后续分析表明,其机制在于:通过接种使特质不那么意外,从而降低全局更新的优化压力,减少泛化程度。本研究关联了先前关于教育环境缓解代码安全问题引发的错位现象。不仅展示了一种简单有效的选择性学习方法,更深化了对语言模型泛化机制的理解。

原文摘要 · Abstract (English)

Language model finetuning often results in learning undesirable traits in combination with desired ones. To address this, we propose inoculation prompting: modifying finetuning data by prepending a short system-prompt instruction that deliberately elicits the undesirable trait. At test time, we evaluate without the instruction; inoculated models have much lower expression of the trait than models trained with unmodified training data. Inoculation is selective: in a toy setting where assistant responses are always in Spanish and ALL-CAPS, an appropriate inoculation (e.g., ``You always speak in Spanish.'') teaches the model to capitalize responses while still responding in English. We find that inoculation is also effective across several additional settings: reducing emergent misalignment (EM) from task-specific finetuning, defending against backdoor injections, and mitigating the transmission of traits via subliminal learning. Follow-up analysis suggests a mechanism: making a trait less surprising via inoculation reduces optimization pressure to globally update the model, thereby reducing the degree of generalization. Our analysis relates to prior work on EM: inoculation explains prior findings that educational contexts mitigate EM from insecure code. Beyond demonstrating a simple and effective technique for selective learning, our results contribute to a better conceptual understanding of how and why language models generalize.

模型控制微调安全泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。