提出新方法让大模型更安全，不丢掉原本的聪明才智。
Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
- 将安全训练梯度投影到通用能力的零空间，避免干扰原有技能。
- 仅用40%安全数据即可达到顶尖安全性能，且通用任务准确率不降。
- 适合追求高效安全对齐的大模型部署场景，尤其看重性能稳定者。
随着大语言模型在现实应用中日益普及，确保其行为符合人类价值观、社会规范和伦理原则至关重要。然而，强化学习中的安全对齐常导致模型遗忘已掌握的通用能力，即所谓的对齐代价。为此，本文提出零空间约束策略优化（NSPO），一种在保持核心能力的同时实现大模型安全对齐的新强化学习框架。通过将安全策略梯度几何地投影至通用任务的零空间，有效缓解对齐代价。理论证明，NSPO能保留模型原始核心能力，同时保证有效的安全对齐下降方向。大量实验表明，NSPO显著优于现有方法，在数学、代码生成和指令遵循等通用任务上保持高精度的同时，实现了最先进的安全表现。值得注意的是，该方法数据高效，仅需PKU-SafeRLHF公开标注安全数据的40%即可获得良好安全性能，无需大量混合通用任务数据，突破了现有对齐方法的资源瓶颈。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) are increasingly deployed in real-world applications, it is important to ensure their behaviors align with human values, societal norms, and ethical principles. However, safety alignment under Reinforcement Learning (RL) often suffers from forgetting learned general abilities, which is also known as the alignment tax. To address this issue, we introduce Null-Space constrained Policy Optimization (NSPO), a novel RL framework for LLM safety alignment while preserving their core abilities. The safety policy gradients are geometrically projected into the null space of general tasks, thereby mitigating the safety alignment tax. In addition, we theoretically prove that NSPO preserves the model's original core capabilities, while still guaranteeing a descent direction for effective safety alignment. Extensive experiments demonstrate that NSPO outperforms existing methods by a large margin, achieving state-of-the-art safety performance without sacrificing accuracy on general tasks, including math, code, and instruction-following tasks. Notably, NSPO is data-efficient and only requires 40% of public human-annotated safety data from PKU-SafeRLHF to achieve promising safety performance, without a large amount of mixed general tasks data in existing alignment methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。