通过抑制有害梯度影响，防护大模型微调中的安全攻击
Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence
- 先将模型置于对有害样本不敏感的平坦损失区域，增强安全对齐
- 微调时按样本类型加权，抑制有害数据影响，提升良性数据学习
- 有效防御恶意微调攻击，适合部署在第三方微调服务场景
微调即服务模式下，当服务提供商使用用户提交的含毒数据集进行模型微调时，可能引发有害微调攻击，威胁大模型安全。本文提出 Antibody 防御策略：首先在微调前使模型处于对有害样本不敏感的平坦损失区域，增强安全对齐鲁棒性；随后在微调阶段采用基于样本加权的算法，抑制有害样本的梯度贡献，同时促进良性样本的学习。实验表明，Antibody 能有效缓解有害微调攻击，同时提升在用户提交数据集上的微调性能。
原文摘要 · Abstract (English)
Fine-tuning-as-a-service introduces a threat to Large Language Models' safety when service providers fine-tune their models on poisoned user-submitted datasets, a process known as harmful fine-tuning attacks. In this work, we show that by regularizing the gradient contribution of harmful samples encountered during fine-tuning, we can effectively mitigate the impact of harmful fine-tuning attacks. To this end, we introduce Antibody, a defense strategy that first ensures robust safety alignment for the model before fine-tuning, and then applies a safety-preservation learning algorithm during fine-tuning. Specifically, in the alignment stage before fine-tuning, we propose optimizing the model to be in a flat loss region with respect to harmful samples, which makes the safety alignment more resilient to subsequent harmful fine-tuning. Then, in the fine-tuning stage, we design a fine-tuning algorithm that applies a weighting scheme to all samples in each training batch to inhibit the model from learning from harmful samples while encouraging learning from benign samples. Experimental results demonstrate that Antibody successfully mitigates harmful fine-tuning attacks while boosting fine-tuning performance on the user-submitted dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。