让大模型抗住恶意微调,还能保持高能力。
AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs
- 用双层优化对抗恶意权重修改,防御者学着抵消攻击。
- 对52种攻击测试,鲁棒性提升最高27.4%,性能损失小于0.5%。
- 适合想开源模型又怕被滥用的研究者和开发者。
开放权重大语言模型的发布在推动研究普及与防止滥用之间形成张力,例如恶意微调可能诱导有害内容输出。现有安全措施难以在保留模型通用能力的同时抵御具备完整权重与架构访问权限的对手,后者可使用全参数微调彻底清除原有防护。为此,我们提出AntiDote,一种用于训练抗篡改大模型的双层优化方法。该方法引入一个辅助对抗超网络,根据防御者模型内部激活状态生成恶意低秩适配(LoRA)权重;防御者模型则通过目标函数学习抵消这些对抗性权重的影响,从而维持其安全对齐。我们在涵盖52种红队攻击的多样化测试中验证了该方法,包括越狱提示、隐空间操纵和直接权重空间攻击。相比抗篡改与遗忘基线,AntiDote在对抗攻击上的鲁棒性最高提升27.4%。关键的是,这一鲁棒性提升仅带来极小的实用性损失,在MMLU、HellaSwag和GSM8K等能力基准上性能下降不超过0.5%。本工作提供了一种实用且计算高效的开放权重模型构建方案,使安全性成为更内嵌、更坚韧的属性。
原文摘要 · Abstract (English)
The release of open-weight large language models (LLMs) creates a tension between advancing accessible research and preventing misuse, such as malicious fine-tuning to elicit harmful content. Current safety measures struggle to preserve the general capabilities of the LLM while resisting a determined adversary with full access to the model's weights and architecture, who can use full-parameter fine-tuning to erase existing safeguards. To address this, we introduce AntiDote, a bi-level optimization procedure for training LLMs to be resistant to such tampering. AntiDote involves an auxiliary adversary hypernetwork that learns to generate malicious Low-Rank Adaptation (LoRA) weights conditioned on the defender model's internal activations. The defender LLM is then trained with an objective to nullify the effect of these adversarial weight additions, forcing it to maintain its safety alignment. We validate this approach against a diverse suite of 52 red-teaming attacks, including jailbreak prompting, latent space manipulation, and direct weight-space attacks. AntiDote is upto 27.4\% more robust against adversarial attacks compared to both tamper-resistance and unlearning baselines. Crucially, this robustness is achieved with a minimal trade-off in utility, incurring a performance degradation of upto less than 0.5\% across capability benchmarks including MMLU, HellaSwag, and GSM8K. Our work offers a practical and compute efficient methodology for building open-weight models where safety is a more integral and resilient property.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。