arXiv:2508.09224cs.CYcs.AI2025-08被引 58

让AI更安全地生成回答,而非简单拒绝用户请求。

From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training

  • 以输出安全性为核心,不再只判断用户意图是否恶意。
  • 在双用途场景下,安全回复率提升,错误响应更轻微。
  • 适合需要平衡安全与帮助性的实际应用开发者。

用于ChatGPT的大语言模型传统上通过学习拒绝边界来训练:根据用户意图,模型被教导要么完全配合,要么直接拒绝。虽然这对明确恶意提示有较强缓解作用,但聚焦于拒绝会导致在用户意图隐蔽时表现脆弱。二元拒绝边界尤其不适用于生物、网络安全等双用途场景,这些场景中高层级回答可安全,但具体细节可能被滥用。为此,我们提出安全补全(safe-completions):一种以助手输出安全性为中心的安全训练方法,而非对用户意图进行二分类。该方法在满足安全政策前提下最大化帮助性。我们将此方法应用于GPT-5,在生产环境对比和内部对照实验中均发现,安全补全训练显著提升了安全性(尤其在双用途提示上),降低了残余安全失败的严重性,并大幅提高模型帮助性。

原文摘要 · Abstract (English)

Large Language Models used in ChatGPT have traditionally been trained to learn a refusal boundary: depending on the user's intent, the model is taught to either fully comply or outright refuse. While this is a strong mitigation for explicitly malicious prompts, focusing safety training on refusals can lead to brittleness for prompts with obscured user intent. Binary refusal boundaries are especially ill-suited for dual-use cases (such as biology or cybersecurity), where a user request can be answered safely at a high level, but in some cases can lead to malicious uplift if sufficiently detailed or actionable. As an alternative, we propose safe-completions: a safety-training approach that centers on the safety of the assistant's output, rather than a binary classification of the user's intent. Safe-completions seek to maximize helpfulness within the safety policy's constraints. We incorporated this approach into GPT-5 and find that across both production comparisons and internally controlled experiments, safe-completion training improves safety (especially on dual-use prompts), reduces the severity of residual safety failures, and substantially increases model helpfulness.

安全训练输出中心双用途

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。