用稀疏自编码器拆解模型特征,让安全对齐更精准高效
Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
- 通过SAE提取单一语义特征,构建可解释的安全子空间
- 在Gemma-2-9B上达99.6%安全率,超越全参数微调7.4个百分点
- 仅更新0.19%参数,适合资源受限场景下的安全对齐
安全对齐——训练大语言模型拒绝有害请求同时保持有用性——对负责任部署至关重要。已有研究发现安全行为受低秩结构支配,暗示参数高效微调(PEFT)应适用于对齐。然而,低秩适配(LoRA)在安全基准测试中始终表现逊于全参数微调和强化学习。我们归因于语义纠缠:由于多义性,安全相关方向与无关概念交织,阻碍隐式子空间识别。为此,我们提出SAILS(基于可解释低秩子空间的安全对齐),利用稀疏自编码器(SAEs)将表示解耦为单义特征,从SAE解码器方向构建可解释的安全子空间,并用于初始化LoRA适配器。理论上,我们证明在单义性假设下,基于SAE的识别可实现任意小的恢复误差,而直接识别存在不可消除的误差下限。实验上,SAILS在Gemma-2-9B上达到最高99.6%安全率,优于全微调7.4个百分点,媲美基于RLHF的模型,同时仅更新0.19%参数并提供可解释性。
原文摘要 · Abstract (English)
Safety alignment -- training large language models (LLMs) to refuse harmful requests while remaining helpful -- is critical for responsible deployment. Prior work established that safety behaviors are governed by low-rank structures, suggesting parameter-efficient fine-tuning (PEFT) should be well-suited for alignment. However, Low-Rank Adaptation (LoRA) consistently underperforms full fine-tuning and reinforcement learning on safety benchmarks. We attribute this gap to semantic entanglement: safety-relevant directions are intertwined with unrelated concepts due to polysemanticity, impeding implicit subspace identification. To address this, we propose SAILS (Safety Alignment via Interpretable Low-rank Subspace), which leverages Sparse Autoencoders (SAEs) to disentangle representations into monosemantic features, constructs an interpretable safety subspace from SAE decoder directions, and uses it to initialize LoRA adapters. Theoretically, we prove that SAE-based identification achieves arbitrarily small recovery error under monosemanticity assumptions, while direct identification suffers an irreducible error floor. Empirically, SAILS achieves up to 99.6% safety rate on Gemma-2-9B -- exceeding full fine-tuning by 7.4 points and matching RLHF-based models -- while updating only 0.19% of parameters and providing interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。