让大模型自己说出训练后的行为，便于安全审计。
Introspection Adapters: Training LLMs to Report Their Learned Behaviors
- 用植入行为微调多个模型，训练单一自省适配器
- 能识别未见过的微调行为，检测隐匿风险达顶尖水平
- 适合模型开发者、安全研究人员用于合规审计
当开发者或用户微调大语言模型时，可能引入意外、恶意或难以察觉的行为。若模型能用自然语言描述自身行为，将极大简化审计流程。本文提出一种可扩展的方法：基于共享基础模型 $M$，通过植入行为 $b_i$ 微调多个模型 $M_i$，形成 $(M_i, b_i)$ 标注数据；再训练一个自省适配器（IA），即单一 LoRA 适配器，在所有微调模型上联合训练，使其能口头表述所学行为。实验发现，该 IA 可泛化至与原始训练方式不同的微调模型，例如在 AuditBench 上实现当前最佳性能，有效识别显式隐藏的危险行为；还可检测加密微调接口攻击。该方法随模型规模和训练数据多样性提升而表现更优。结果表明，自省适配器是一种高效、可扩展且实用的微调模型审计方案。
原文摘要 · Abstract (English)
When model developers or users fine-tune an LLM, this can induce behaviors that are unexpected, deliberately harmful, or hard to detect. It would be far easier to audit LLMs if they could simply describe their behaviors in natural language. Here, we study a scalable approach to rapidly identify learned behaviors of many LLMs derived from a shared base LLM. Given a model $M$, our method works by finetuning models $M_i$ from $M$ with implanted behaviors $b_i$; the $(M_i, b_i)$ pairs serve as labeled training data. We then train an introspection adapter (IA): a single LoRA adapter jointly trained across the finetunes $M_i$ to cause them to verbalize their implanted behaviors. We find that this IA induces self-description of learned behaviors even in finetunes of $M$ that were trained in very different ways from the $M_i$. For example, IAs generalize to AuditBench, achieving state-of-the-art at identifying explicitly hidden concerning behaviors. IAs can also be used to detect encrypted finetuning API attacks. They scale favorably with model size and training data diversity. Overall, our results suggest that IAs are a scalable, effective, and practically useful approach to auditing fine-tuned LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。