让大模型自我推理来守护安全，效果超越GPT-4
GuardReasoner: Towards Reasoning-based LLM Safeguards
- 用详细推理步骤训练守卫模型，提升判断能力
- 在13个任务上平均F1得分超LLaMA Guard 3 20.84%
- 适合需要可解释安全防护的AI应用开发者
随着大语言模型在安全关键场景中广泛应用，如何通过护栏保障其安全性仍是核心挑战。本文提出GuardReasoner，通过引导守卫模型学习推理来增强安全防护能力。具体而言，构建了包含127,000样本、460,000条推理步骤的GuardReasonerTrain数据集；引入推理SFT以激活守卫模型的推理能力；并提出硬样本DPO进一步强化其推理表现。实验在13个基准、3类护栏任务上验证了该方法的优越性：GuardReasoner 8B在平均F1得分上超过GPT-4o+CoT 5.74%，较LLaMA Guard 3 8B提升20.84%。项目代码、训练数据及多尺度模型（1B、3B、8B）已开源。
原文摘要 · Abstract (English)
As LLMs increasingly impact safety-critical applications, ensuring their safety using guardrails remains a key challenge. This paper proposes GuardReasoner, a new safeguard for LLMs, by guiding the guard model to learn to reason. Concretely, we first create the GuardReasonerTrain dataset, which consists of 127K samples with 460K detailed reasoning steps. Then, we introduce reasoning SFT to unlock the reasoning capability of guard models. In addition, we present hard sample DPO to further strengthen their reasoning ability. In this manner, GuardReasoner achieves better performance, explainability, and generalizability. Extensive experiments and analyses on 13 benchmarks of 3 guardrail tasks demonstrate its superiority. Remarkably, GuardReasoner 8B surpasses GPT-4o+CoT by 5.74% and LLaMA Guard 3 8B by 20.84% F1 score on average. We release the training data, code, and models with different scales (1B, 3B, 8B) of GuardReasoner : https://github.com/yueliu1999/GuardReasoner/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。