用持续学习防止大模型微调后安全能力退化。
Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning
- 将安全对齐问题建模为持续学习，避免遗忘原有安全特性。
- 在三种模型、三类任务上，攻击成功率显著低于标准微调。
- 适合需要长期安全稳定的模型定制服务场景。
随着大语言模型的普及，其安全对齐愈发重要。本文研究了模型在适应新任务时出现的安全性能退化问题，将其归因于灾难性遗忘，并将保护微调过程中的安全对齐视为持续学习（CL）问题。在微调即服务的设定下，用户上传数据以获取定制化模型。我们评估了多种来自文献的持续学习方法，包括基于正则化、记忆存储和模型融合的方法，在良性与恶意用户数据两种场景下进行实验。结果表明，所有持续学习方法均显著降低攻击成功率，其中DER方法表现最优，优于其他持续学习方法及现有安全保护基线。该结论在GSM8K、SST2、Code三个下游任务和LLaMA2-7B、Mistral-7B、Gemma-2B三个模型家族中均成立，验证了持续学习作为安全对齐实用解决方案的有效性。
原文摘要 · Abstract (English)
The safety alignment of large language models (LLMs) is becoming increasingly important with their democratization. In this paper, we study the safety degradation that comes with adapting LLMs to new tasks. We attribute this safety compromise to catastrophic forgetting and frame the problem of preserving safety when fine-tuning as a continual learning (CL) problem. We consider the fine-tuning-as-a-service setup where the user uploads their data to a service provider to get a customized model that excels on the user's selected task. We adapt several CL approaches from the literature and systematically evaluate their ability to mitigate safety degradation. These include regularization-based, memory-based, and model merging approaches. We consider two scenarios, (1) benign user data and (2) poisoned user data. Our results demonstrate that CL approaches consistently achieve lower attack success rates than standard fine-tuning. Among these, DER outperforms both other CL methods and existing safety-preserving baselines while maintaining task utility. These findings generalize across three downstream tasks (GSM8K, SST2, Code) and three model families (LLaMA2-7B, Mistral-7B, Gemma-2B), establishing CL as a practical solution to preserve safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。