arXiv:2505.16737cs.LGcs.AI2025-05被引 10

通过安全感知探针,在微调中防止模型生成有害内容。

Secure LLM Fine-Tuning via Safety-Aware Probing

  • 用对比安全信号定位影响安全的隐藏状态方向。
  • 微调时动态调整参数更新路径,提升安全性同时保持任务性能。
  • 适合关注大模型安全微调的研究者和应用开发者。

大语言模型在众多应用中取得显著成果,但其生成有害内容的能力引发严重安全担忧。尽管预训练或后训练阶段常采用安全对齐技术,但近期研究表明,后续在对抗性或良性数据上微调仍可能损害模型安全性。本文重新审视为何在非有害数据上微调仍会降低安全性:发现安全与任务性能的损失景观部分解耦,优化任务性能的更新可能使模型滑向不安全区域。基于此,提出安全感知探针(SAP)框架,在微调过程中利用对比安全信号识别安全相关方向,并通过轻量级探针扰动隐藏状态传播,引导参数更新避开有害轨迹,同时保留任务学习能力。大量实验表明,SAP在多个模型与任务上均显著改善安全-效用权衡。平均而言,相较于标准微调,SAP显著降低有害得分,优于强基线且保持竞争力的任务性能。SAP在有害数据投毒、对抗性微调及专用后微调自适应攻击下也表现出更强鲁棒性,验证其在微调阶段保障LLM安全的有效性与可扩展性。代码已开源。

原文摘要 · Abstract (English)

Large language models (LLMs) have achieved remarkable success across many applications, but their ability to generate harmful content raises serious safety concerns. Although safety alignment techniques are often applied during pre-training or post-training, recent studies show that subsequent fine-tuning on adversarial or even benign data can still compromise model safety. In this paper, we revisit the fundamental question of why fine-tuning on non-harmful data may nevertheless degrade safety. We show that the safety and task-performance loss landscapes are partially decoupled, so updates that improve task-specific performance may still move the model toward unsafe regions. Based on this insight, we propose a safety-aware probing (SAP) optimization framework for mitigating safety risks during fine-tuning. Concretely, SAP uses contrastive safety signals to locate safety-correlated directions, and optimizes a lightweight probe that perturbs hidden-state propagation during fine-tuning, thereby steering parameter updates away from harmful trajectories while preserving task-specific learning. Extensive experiments show that SAP consistently improves the safety--utility tradeoff across multiple models and tasks. Averaged over multiple LLMs, SAP reduces the harmful score significantly relative to standard fine-tuning, outperforming strong baselines while maintaining competitive task-specific performance. SAP also demonstrates stronger robustness under harmful data poisoning, adversarial fine-tuning, and a dedicated post-fine-tuning adaptive attack, validating that SAP is an effective and scalable framework for preserving LLM safety during fine-tuning. Our code is available at https://github.com/ChengcanWu/SAP.

大模型安全微调探针机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。