微调后语言模型易出安全问题,该方法不修改模型就能自动修复。
HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

- 通过激活指纹生成专属安全模块,不改动原模型权重。
- 在多个测试集上将有害回复率从19%-31%降至1%以下。
- 适合已部署的微调模型,无需额外训练或数据。
大型语言模型在微调后可能出现安全对齐脆弱的问题,即使良性任务适配也可能导致有害响应增加。现有防御方法主要分两类:一类在微调过程中或之后通过重训练或权重修改干预,成本高且可能损害任务性能;另一类使用与模型无关的安全分类器,但可能遗漏特定微调版本的特异性风险。为克服这些局限,我们提出 HyperSafe,一种事后、模型专有、非侵入式的安全恢复框架。HyperSafe 为每个微调后的检查点生成一个模型专属的安全侧网络(SSN),利用层级激活指纹捕捉微调带来的内部表示变化。通过少量校准提示,超网络可在单次前向传播中将这些指纹映射为 SSN 的参数。生成的 SSN 与冻结的微调模型并行运行,进行提示级安全分类:有害提示被引导拒绝,安全提示由原模型回答。因此,HyperSafe 不需要梯度更新、部署时无需安全数据,也不修改部署模型权重。我们在 Qwen2-7B 与 LLaMA-3-8B 两个模型族上评估,所有未见检查点的有害响应率均从 19%-31% 降低至 1% 以下,同时下游任务准确率平均仅比微调基线低 1%。代码已开源。
原文摘要 · Abstract (English)
Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task performance, or they use model-agnostic safety classifiers, which may miss failures specific to a given fine-tuned checkpoint. These limitations motivate a post hoc, model-specific, and non-invasive approach to safety restoration. To meet these requirements, we propose HyperSafe, a framework that restores safety behavior by generating a model-specific Safe Side Network (SSN) for each fine-tuned checkpoint. HyperSafe uses layer-wise activation fingerprints to capture how fine-tuning changes the model's inner representations. With a small set of given calibration prompts, the hypernetwork maps these fingerprints to the parameters of the \ssn{} in a single forward pass. The generated \ssn{} runs alongside the frozen fine-tuned model and performs prompt-level safety classification: harmful prompts are routed to refusal, while safe prompts are answered by the original fine-tuned model. Thus, HyperSafe requires no gradient updates, no safety data at deployment time, and no modification to the deployed model weights. We evaluate HyperSafe on two model families, Qwen2-7B and LLaMA-3-8B, across multiple safety benchmarks. HyperSafe reduces harmful response rates from 19-31% to below 1% on every held-out checkpoint, while keeping downstream task accuracy within 1% of the fine-tuned baseline on average. Code is available at https://github.com/nokronim/project-safety-remedy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。