通过低秩安全子空间融合,修复微调后模型的安全性,不影响原有能力。
LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion
- 构建低秩投影矩阵提取安全向量主成分,实现安全信息分离。
- 修复后模型在多个基准上安全率提升超20%,下游任务性能损失<1%。
- 适用于需保持性能的模型安全加固,如医疗、金融场景部署。
大语言模型的安全机制存在显著脆弱性,即使在无有害内容的数据集上微调,也可能削弱其安全性。现有对齐方法多依赖微调过程,导致复杂度和计算开销上升。本文提出LSSF(低秩安全子空间融合)框架,利用大模型中安全信息的低秩特性,通过构建低秩投影矩阵提取安全向量的主成分。该投影矩阵代表模型的稳定安全子空间,在微调过程中保持不变,且与通用能力解耦。将这些主成分通过线性组合与微调后的模型结合,可有效恢复安全性。此外,为应对不同层间安全信息编码密度差异,提出安全奇异值熵指标,动态计算每层的安全关键秩。大量实验表明,该后处理对齐方法能有效恢复微调模型的安全性,对下游任务性能影响极小。
原文摘要 · Abstract (English)
The safety mechanisms of large language models (LLMs) exhibit notable fragility, as even fine-tuning on datasets without harmful content may still undermine their safety capabilities. Meanwhile, existing safety alignment methods predominantly rely on the fine-tuning process, which inadvertently leads to the increased complexity and computational resources required. To address these issues, we introduce LSSF, a novel safety re-alignment framework with \underline{L}ow-Rank \underline{S}afety \underline{S}ubspace \underline{F}usion. Our proposed method exploits the low-rank characteristics of safety information in LLMs by constructing a low-rank projection matrix to extract the principal components of safety vectors. Notably, this projection matrix represents the low-rank safety subspace of the LLMs, which we have observed to remain stable during fine-tuning process and is isolated from the model's general capabilities. These principal components are used to effectively restore safety alignment when combined with fine-tuned LLMs through linear arithmetic. Additionally, to account for the varying encoding densities of safety information across different layers of LLMs, we propose a novel metric called safety singular value entropy. This metric quantifies the encoding density and allows for the dynamic computation of the safety-critical rank for each safety vector. Extensive experiments demonstrate that our proposed post-hoc alignment method can effectively restore the safety alignment of fine-tuned models with minimal impact on their performance in downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。