用案例增强推理,让大模型更安全且不误拒正常请求
Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety
- 用真实案例辅助推理,替代繁琐的规则条文
- 在多个测试中提升安全性,同时减少误拒率
- 适合需要平衡安全与实用性的大模型应用
确保大语言模型遵守安全原则而不拒绝无害请求仍是重大挑战。尽管OpenAI通过推理详细‘代码式’安全规则实现了其o系列模型的推理对齐(DA),但该方法在缺乏先进推理能力的开源模型中的有效性尚不明确。本文系统评估了显式指定大量安全规则与通过示例案例展示规则的效果差异。结果表明,直接引用规则会不一致地提升无害性,但系统性降低帮助性;而基于案例增强的简化规则训练则能实现更稳健、泛化性更强的安全行为。为此,我们提出CADA——一种基于自生成安全推理链的强化学习方法,通过案例增强推理来引导模型。CADA有效提升了无害性,增强了对抗攻击的鲁棒性,并减少了过度拒绝,同时在多种基准上保持了良好实用性,为提升模型安全性和帮助性提供了一种切实可行的替代方案。
原文摘要 · Abstract (English)
Ensuring that Large Language Models (LLMs) adhere to safety principles without refusing benign requests remains a significant challenge. While OpenAI introduces deliberative alignment (DA) to enhance the safety of its o-series models through reasoning over detailed ``code-like'' safety rules, the effectiveness of this approach in open-source LLMs, which typically lack advanced reasoning capabilities, is understudied. In this work, we systematically evaluate the impact of explicitly specifying extensive safety codes versus demonstrating them through illustrative cases. We find that referencing explicit codes inconsistently improves harmlessness and systematically degrades helpfulness, whereas training on case-augmented simple codes yields more robust and generalized safety behaviors. By guiding LLMs with case-augmented reasoning instead of extensive code-like safety rules, we avoid rigid adherence to narrowly enumerated rules and enable broader adaptability. Building on these insights, we propose CADA, a case-augmented deliberative alignment method for LLMs utilizing reinforcement learning on self-generated safety reasoning chains. CADA effectively enhances harmlessness, improves robustness against attacks, and reduces over-refusal while preserving utility across diverse benchmarks, offering a practical alternative to rule-only DA for improving safety while maintaining helpfulness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。