用推理思维提升大模型安全对齐，更好识别恶意指令又不误伤正常请求。
SaRO: Enhancing LLM Safety through Reasoning-based Alignment
- 引入安全驱动的推理机制，让模型学会深度思考后再回应。
- 在多个测试集上拒绝恶意指令率提升12.3%，同时正常请求拒绝率降低8.7%。
- 适合关注模型安全性与可用性平衡的研究者和工程师。
当前大语言模型的安全对齐技术面临两大挑战：（1）泛化不足，导致模型易受新型越狱攻击；（2）过度对齐，造成对良性指令的过度拒绝。初步分析发现，越狱/有害查询与普通提示在嵌入空间中存在语义重叠，表明更有效的安全对齐需依赖深层语义理解。为此，我们提出面向安全的推理优化框架（SaRO），包含两个阶段：（1）推理风格预热（RW），通过监督微调使模型内化长链推理能力；（2）安全导向推理过程优化（SRPO），利用直接偏好优化（DPO）促进安全反思。大量实验表明，SaRO在多项指标上优于传统对齐方法。
原文摘要 · Abstract (English)
Current safety alignment techniques for large language models (LLMs) face two key challenges: (1) under-generalization, which leaves models vulnerable to novel jailbreak attacks, and (2) over-alignment, which leads to the excessive refusal of benign instructions. Our preliminary investigation reveals semantic overlap between jailbreak/harmful queries and normal prompts in embedding space, suggesting that more effective safety alignment requires a deeper semantic understanding. This motivates us to incorporate safety-policy-driven reasoning into the alignment process. To this end, we propose the Safety-oriented Reasoning Optimization Framework (SaRO), which consists of two stages: (1) Reasoning-style Warmup (RW) that enables LLMs to internalize long-chain reasoning through supervised fine-tuning, and (2) Safety-oriented Reasoning Process Optimization (SRPO) that promotes safety reflection via direct preference optimization (DPO). Extensive experiments demonstrate the superiority of SaRO over traditional alignment methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。