动态调节正则化,让模型微调中始终安全可靠。
Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning
- 根据安全风险自适应调整正则化强度,防止微调时安全性能下降。
- 两种风险估计方法均有效降低攻击成功率,且不损失下游任务性能。
- 适合关注模型安全性的研究人员和工业级大模型部署团队。
指令遵循语言模型在训练中追求有用与安全,但在良性微调下安全性可能退化,对抗性更新更会加剧恶化。现有防御手段常保护有限或需牺牲实用性。本文提出一种动态正则化训练框架,能随安全风险变化调节约束力度,确保模型全程对齐。通过两种方式估算训练时的安全风险:基于裁判的Safety Critic为批次分配高层伤害评分;基于激活的轻量级分类器从中间层激活预测有害意图。高风险更新被约束靠近安全参考策略,低风险则正常训练。实验证明,预生成激活中可预测有害意图,裁判评分提供高召回率的安全指引。在多个模型家族与攻击场景下,该方法相比标准微调显著降低攻击成功率,保持下游性能,且无推理开销。本工作展示了一种兼顾安全与实用性的可扩展机制。
原文摘要 · Abstract (English)
Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates. Existing defenses often offer limited protection or force a trade-off between safety and utility. We introduce a training framework that adapts regularization in response to safety risk, enabling models to remain aligned throughout fine-tuning. To estimate safety risk at training time, we explore two distinct approaches: a judge-based Safety Critic that assigns high-level harm scores to training batches, and an activation-based risk predictor built with a lightweight classifier trained on intermediate model activations to estimate harmful intent. Each approach provides a risk signal that is used to constrain updates deemed higher risk to remain close to a safe reference policy, while lower-risk updates proceed with standard training. We empirically verify that harmful intent signals are predictable from pre-generation activations and that judge scores provide effective high-recall safety guidance. Across multiple model families and attack scenarios, adaptive regularization with either risk estimation approach consistently lowers attack success rate compared to standard fine-tuning, preserves downstream performance, and adds no inference-time cost. This work demonstrates a principled mechanism for maintaining safety without sacrificing utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。