微调后模型易失安全，用分层选择性合并修复
SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
- 按层选择性合并安全模型与微调模型
- 减少有害输出，对任务性能影响极小
- 无需复杂算法，适合实际部署
微调大语言模型（LLMs）常用于适配特定领域，但近期研究表明，微调可能削弱模型的安全对齐，使其对有害或不道德提示产生不当响应。尽管已有多种重对齐安全的方法被提出，但往往引入复杂的定制算法，难以实现且可能损害任务性能。本文提出SafeMERGE，一种轻量级的后微调框架，在保持下游性能的同时恢复安全对齐。该方法仅在特定层偏离安全行为时，通过余弦相似度判断，选择性地合并安全对齐模型的层。在四个LLM和多个任务上，SafeMERGE consistently降低有害输出，对性能影响可忽略甚至为正。结果表明，选择性分层合并能有效防范微调中无意丢失安全性的风险，建立了一种简单而有效的后处理防护机制。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) is a common practice to adapt generalist models to specialized domains. However, recent studies show that fine-tuning can erode safety alignment, causing LLMs to respond to harmful or unethical prompts. Many methods to realign safety have been proposed, but often introduce custom algorithms that are difficult to implement or compromise task utility. In this work, we propose SafeMERGE, a lightweight, post-fine-tuning framework that restores safety while maintaining downstream performance. SafeMERGE selectively merges fine-tuned with safety-aligned model layers only when they deviate from safe behavior, measured by a cosine similarity criterion. Across four LLMs and several tasks, SafeMERGE consistently reduces harmful outputs compared to other defenses, with negligible or even positive impact on utility. Our results demonstrate that selective, layer-wise merging offers a robust safeguard against the inadvertent loss of safety during fine-tuning, establishing SafeMERGE as a simple yet effective post-fine-tuning defense.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。