提出自适应安全上下文学习框架，缓解大模型安全与能力的矛盾。
Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning
- 将安全对齐建模为多轮工具使用过程，让模型自主决定何时调用规则。
- 在多个基准上实现更高推理性能，相比基线提升12.3%的准确率。
- 适合需要高安全性和强推理能力的场景，如医疗、金融决策系统。
尽管推理模型在复杂任务中取得显著进展，其日益增强的能力也要求更严格的安全措施。安全对齐的核心挑战在于安全与实用性之间的固有权衡。然而，现有对齐策略通常通过上下文提炼构建带有显式安全规则的思维链（CoT）训练数据，这无意中通过将规则记忆与拒绝响应绑定，限制了推理能力。为缓解这一权衡，我们提出自适应安全上下文学习（ASCL）框架，以在适当上下文中提升推理能力。ASCL将安全对齐建模为多轮工具使用过程，使模型能自主决定何时调用安全规则并生成后续推理。此外，为应对强化学习中对规则调用的偏好，我们引入逆频率策略优化（IFPO）以重新平衡优势估计。通过解耦规则检索与后续推理，我们的方法在多个基准测试中优于基线，整体性能显著提升。代码已公开于 https://github.com/ybwang119/ASCL。
原文摘要 · Abstract (English)
While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning~(ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization~(IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines. Our code is publicly available at https://github.com/ybwang119/ASCL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。