通过意图推理与查询重写,让大模型更安全地回应敏感请求。
IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement
- 用专门的守护模型分析用户意图并重写潜在危险问题。
- 在16.3万条数据上训练,过拒率降低30%以上,响应质量提升。
- 适合需要高安全又怕误伤的AI应用开发者使用。
大语言模型的快速发展带来了广泛应用,但生成有害内容的安全隐患仍突出。现有防护方法常过度拒绝无害请求,难以平衡安全性、误拒率与实用性。本文提出IntentionReasoner,一种新型防护机制,通过专用守护模型实现意图推理、多层级安全分类与查询重写,以化解边缘案例中的潜在危害。研究构建了约16.3万条标注数据集,包含意图分析、安全标签和重写版本;通过监督微调赋予守护模型格式遵循、意图解析与安全重写能力;再结合规则启发式与奖励模型信号,在强化学习框架中采用定制化多奖励优化策略进一步提升性能。大量实验表明,IntentionReasoner在多个安全基准、生成质量评估及越狱攻击场景中表现优异,显著增强安全性,有效降低过拒率并改善响应质量。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) has driven their adoption across diverse domains, yet their ability to generate harmful content poses significant safety challenges. While extensive research has focused on mitigating harmful outputs, such efforts often come at the cost of excessively rejecting harmless prompts. Striking a balance among safety, over-refusal, and utility remains a critical challenge. In this work, we introduce IntentionReasoner, a novel safeguard mechanism that leverages a dedicated guard model to perform intent reasoning, multi-level safety classification, and query rewriting to neutralize potentially harmful intent in edge-case queries. Specifically, we first construct a comprehensive dataset comprising approximately 163,000 queries, each annotated with intent reasoning, safety labels, and rewritten versions. Supervised fine-tuning is then applied to equip the guard model with foundational capabilities in format adherence, intent analysis, and safe rewriting. Finally, we apply a tailored multi-reward optimization strategy that integrates rule-based heuristics and reward model signals within a reinforcement learning framework to further enhance performance. Extensive experiments show that IntentionReasoner excels in multiple safeguard benchmarks, generation quality evaluations, and jailbreak attack scenarios, significantly enhancing safety while effectively reducing over-refusal rates and improving the quality of responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。