用强化对抗训练提升大模型防恶意微调能力
Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks
- 通过放大攻击优化步数模拟更强的恶意微调
- 相比普通微调，防御效果显著提升
- 适用于多种模型规模和攻击场景
当前开放权重的大语言模型易受恶意微调攻击，仅需少量监督微调（SFT）步骤即可破坏其安全对齐。现有对齐阶段防御方法主要针对参数高效微调攻击，但无法抵御全参数微调攻击。本文提出Patcher方法，受对抗训练与双层优化启发，通过在对抗循环中增加优化步数来强化模拟攻击，迫使防御方找到对强攻击不敏感的模型参数。此外，我们设计了一种高效的并行算法，在保持性能的同时减少训练耗时。大量实验表明，相较于基础SFT对齐，Patcher显著提升了模型鲁棒性，并能泛化至多种攻击场景和模型尺寸。代码已开源：https://github.com/haomingwen/patcher。
原文摘要 · Abstract (English)
Current open-weight large language models (LLMs) are prone to malicious finetuning attacks, which could compromise the safety alignment of LLMs with only a few steps of supervised finetuning (SFT) on poisoned datasets. Existing alignment-stage defenses are primarily designed to defend against attacks that use parameter-efficient finetuning methods. However, they fail to defend against stronger attacks that use full-parameter finetuning. In this paper, we propose Patcher, a method inspired by adversarial training and bi-level optimization, to combat such attacks. Patcher strengthens the simulated attack by scaling up the optimization steps in the adversarial loop, thus forcing the defender to find model parameters that are insensitive to stronger attacks. Furthermore, we propose an efficient parallel algorithm to implement Patcher, decreasing the wall-clock time of training while preserving Patcher's performance. Extensive experiments show that Patcher substantially improves the model's robustness compared to vanilla SFT alignment, and transfers to diverse attack scenarios and model sizes. Code is available at https://github.com/haomingwen/patcher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。