用LoRA微调拒绝数据,既能保推理能力又提升安全,无需牺牲性能。
LoRA is All You Need for Safety Alignment of Reasoning LLMs
- 在SFT阶段用低秩自适应(LoRA)微调拒绝数据集
- 安全性能媲美全模型对齐,推理能力几乎不下降
- 适合需要安全与推理兼顾的LLM部署场景
具备推理能力的大语言模型在解决复杂问题上取得重大突破,但近期研究发现,训练和部署强推理能力可能引入显著安全风险。通常通过在推理学习后增加二次安全对齐阶段来缓解,但该过程常导致推理性能下降,即“安全税”现象。本文提出一种简单有效的方法:在拒绝数据集的SFT阶段应用LoRA。尽管方法简洁,其效果在多个模型规模、架构、两个安全基准和四个推理基准(涵盖数学、科学、代码生成)上均表现优异,实现与全模型对齐相当的安全性,同时保持接近原始推理微调模型的性能。进一步消融实验表明:(1) 阶数为1的更新已足够实现最优安全-推理平衡;(2) 仅更新MLP上投影层优于更新整个MLP;(3) 中间层更新比早/晚层更有效。最后的理论分析揭示,当微调任务的秩超出预算时,基础任务性能退化速率与基础任务内在维度成反比,说明当微调任务低秩而基础能力高秩时,LoRA最为有效。
原文摘要 · Abstract (English)
Reasoning-capable LLMs have achieved major breakthroughs in solving complex problems, but recent work shows that acquiring and deploying strong reasoning can introduce significant safety risks. A common mitigation is to apply a secondary safety-alignment phase after reasoning is learned; however, safety alignment often degrades reasoning performance--a phenomenon known as the "Safety Tax". In this work, we show that a simple approach can largely bypass this trade-off: applying LoRA during SFT on refusal datasets. Despite its simplicity, this recipe achieves safety comparable to full-model alignment while preserving reasoning performance close to the original reasoning-tuned model, and the result holds across multiple model sizes and architectures, two safety benchmarks, and four reasoning benchmarks spanning mathematics, science, and code generation. We further ablate LoRA configurations and find that (1) rank-1 updates are sufficient to achieve the best safety-reasoning trade-off, (2) applying LoRA only to the MLP up-projection layers can outperform updating the full MLP, and (3) updating middle layers is more effective than updating early or late layers. Finally, we provide a theoretical analysis that helps understand when and why LoRA works, revealing that overshooting the rank budget (using a larger rank than needed for the finetuning task) induces base-task degradation at a rate inversely proportional to the intrinsic dimensionality of the base task. This suggests LoRA is most effective when the finetuning task is low-rank and the base capability is high-rank.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。