用生成数据保持大模型安全对齐，防止微调时变“危险”
GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning
- 用大模型自动生成安全对齐数据，替代无法获取的原始数据
- 实验显示能有效减少微调导致的安全性下降，性能几乎不变
- 适合关注模型安全、做下游微调的研究者和工程师
近期研究发现，即使使用看似无害的微调方法，大型语言模型（LLMs）的安全对齐也可能被轻易破坏。为在微调过程中保持安全对齐，常用策略是联合优化安全与任务目标，但原始对齐数据通常对开放权重的LLM也不可访问。受持续学习中生成回放的启发，我们提出生成式回放安全对齐保持框架GR-SAP，通过大模型合成特定领域的对齐数据，并在下游适应过程中整合这些数据以维持安全对齐。理论与实证分析表明，这些合成数据可作为原始对齐数据的可靠代理。在多种模型和下游任务上的实验显示，GR-SAP显著缓解了微调引发的安全性退化，同时保持了相当的下游性能。代码已开源：https://github.com/chili-lab/gr-sap。
原文摘要 · Abstract (English)
Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs. Inspired by generative replay in continual learning, we propose Generative Replay for Safety Alignment Preservation (GR-SAP), a unified framework that synthesizes domain-specific alignment data from LLMs and integrate them during downstream adaption to preserve safety alignment. Theoretical and empirical analyses demonstrate this synthetic data serves as a reliable proxy for the original alignment data. Experiments across various models and downstream tasks show that GR-SAP substantially mitigates fine-tuning-induced safety degradation while maintaining comparable downstream performance. Our code is available at https://github.com/chili-lab/gr-sap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。