让大模型学会按安全指南推理，提升对未知攻击的防御能力。
Safety Reasoning with Guidelines
- 通过合成符合安全指南的推理监督信号，引导模型深度思考。
- BoN实验显示，随着N增大，安全性能显著提升，说明模型有隐藏安全知识。
- 适合关注大模型安全、对抗攻击防御的研究者和开发者。
训练安全的大语言模型仍是关键挑战。目前最常用的方法——拒绝训练（Refusal Training, RT）在面对各类分布外（OOD）越狱攻击时难以泛化。尽管已有多种先进方法提出，我们却质疑：是否OOD攻击本质上已超出基础RT的能力？通过最佳候选数（Best-of-N, BoN）评估发现，随着N增加，安全性能显著提升，表明模型具备充分的潜在安全知识，但RT无法在OOD场景下持续激发这些知识。进一步的领域适应分析揭示，直接训练会导致模型依赖表面捷径，形成非泛化的表征映射。受此启发，我们提出让模型针对每个查询进行安全推理。具体而言，我们合成与指定安全指南一致的推理监督信号，反映多样化的安全认知视角。这促使模型开展深层推理，显式激活并利用潜藏的安全知识。大量实验表明，该方法显著提升了模型对分布外攻击的泛化能力。
原文摘要 · Abstract (English)
Training safe LLMs remains a critical challenge. The most widely used method, Refusal Training (RT), struggles to generalize against various Out-of-Distribution (OOD) jailbreaking attacks. Although various advanced methods have been proposed to address this issue, we instead question whether OOD attacks inherently surpass the capability of vanilla RT. Evaluations using Best-of-N (BoN) reveal significant safety improvements as N increases, indicating models possess adequate latent safety knowledge but RT fails to consistently elicit it under OOD scenarios. Further domain adaptation analysis reveals that direct RT causes reliance on superficial shortcuts, resulting in non-generalizable representation mappings. Inspired by our findings, we propose training model to perform safety reasoning for each query. Specifically, we synthesize reasoning supervision aligned with specified guidelines that reflect diverse perspectives on safety knowledge. This encourages model to engage in deeper reasoning, explicitly eliciting and utilizing latent safety knowledge for each query. Extensive experiments show that our method significantly improves model generalization against OOD attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。