微调会意外唤醒被屏蔽的有害内容，新方法可防止这一风险。
Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models
- 分离训练安全模块与微调模块，推理时合并
- 在无害数据上微调后仍能防止有害内容重现
- 适合关注生成模型安全性的研究人员和开发者
微调文本到图像扩散模型广泛用于个性化和新领域适应。本文揭示了微调的一个关键漏洞：旨在过滤有害内容（如裸露）的安全对齐方法在微调过程中可能失效，导致先前被抑制的内容重新出现，即使使用无害数据集也是如此。尽管大语言模型中存在此类‘微调越狱’问题，但在文本到图像扩散模型中尚未被充分研究。我们的研究发现，标准微调可能无意间破坏安全措施，使模型重新学习已被移除的有害概念，甚至加剧有害行为。为此，我们提出一种新颖且立即可行的解决方案——模块化LoRA：将安全低秩适配（LoRA）模块与微调LoRA组件分开训练，并在推理时合并。该方法有效防止有害内容的重学，同时不损害模型在新任务上的性能。实验表明，模块化LoRA在维持安全对齐方面优于传统微调方法，为提升文本到图像扩散模型安全性提供了实用方案。
原文摘要 · Abstract (English)
Fine-tuning text-to-image diffusion models is widely used for personalization and adaptation for new domains. In this paper, we identify a critical vulnerability of fine-tuning: safety alignment methods designed to filter harmful content (e.g., nudity) can break down during fine-tuning, allowing previously suppressed content to resurface, even when using benign datasets. While this "fine-tuning jailbreaking" issue is known in large language models, it remains largely unexplored in text-to-image diffusion models. Our investigation reveals that standard fine-tuning can inadvertently undo safety measures, causing models to relearn harmful concepts that were previously removed and even exacerbate harmful behaviors. To address this issue, we present a novel but immediate solution called Modular LoRA, which involves training Safety Low-Rank Adaptation (LoRA) modules separately from Fine-Tuning LoRA components and merging them during inference. This method effectively prevents the re-learning of harmful content without compromising the model's performance on new tasks. Our experiments demonstrate that Modular LoRA outperforms traditional fine-tuning methods in maintaining safety alignment, offering a practical approach for enhancing the security of text-to-image diffusion models against potential attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。