提出Booster方法，缓解大模型微调时的有害扰动风险
Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation
- 在对齐阶段引入正则化损失，抑制权重扰动带来的有害影响
- 实测可降低微调后模型的有害得分，同时保持下游任务性能
- 适合关注模型安全微调的开发者与研究人员
有害微调攻击对大型语言模型的微调即服务模式构成严重安全威胁。尽管已有防御方案被提出，但其效果仍不理想，且问题根源尚未完全揭示。本文指出，模型权重上的有害扰动可能是导致对齐失效的潜在原因。为此，我们提出一种对齐阶段解决方案——Booster。技术上，在原始对齐损失基础上，添加一个正则化损失，确保模型在模拟有害扰动后，有害损失的下降幅度被削弱，从而降低后续微调的风险。实验结果表明，Booster能有效降低微调后模型的有害得分，同时保持下游任务性能。代码已开源：https://github.com/git-disl/Booster。
原文摘要 · Abstract (English)
Harmful fine-tuning attack poses serious safety concerns for large language models' fine-tuning-as-a-service. While existing defenses have been proposed to mitigate the issue, their performances are still far away from satisfactory, and the root cause of the problem has not been fully recovered. To this end, we in this paper show that harmful perturbation over the model weights could be a probable cause of alignment-broken. In order to attenuate the negative impact of harmful perturbation, we propose an alignment-stage solution, dubbed Booster. Technically, along with the original alignment loss, we append a loss regularizer in the alignment stage's optimization. The regularizer ensures that the model's harmful loss reduction after the simulated harmful perturbation is attenuated, thereby mitigating the subsequent fine-tuning risk. Empirical results show that Booster can effectively reduce the harmful score of the fine-tuned models while maintaining the performance of downstream tasks. Our code is available at https://github.com/git-disl/Booster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。