在推理过程中动态插入安全提醒，防止大模型生成有害内容。
ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
- 通过分析模型内部注意力机制，识别关键推理节点。
- 在推理中注入安全反思，有效抵御四类越狱攻击。
- 无需微调，低开销，适合实际部署的防护方案。
大型推理模型在复杂任务中表现优异，但在推理中后期仍易生成有害内容。现有防御方法依赖昂贵的微调和专家知识，难以扩展。本文提出ReasoningGuard，一种推理时的安全防护机制。它利用模型内部注意力机制精准定位推理路径中的关键节点，在适当时刻触发安全反思，引导模型走向无害且有帮助的推理方向。为同时保护后续推理步骤与最终答案，采用缩放采样策略在解码阶段选择最优路径。该方法仅带来少量额外推理开销，有效缓解四类越狱攻击，包括针对推理过程的新攻击方式。相比九种现有防护手段，ReasoningGuard表现更优，实现当前最佳防御效果，同时避免常见过度保守的安全问题。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have demonstrated impressive performance in reasoning-intensive tasks, but they remain vulnerable to harmful content generation, particularly in the mid-to-late steps of their reasoning processes. Current defense methods, however, depend on costly fine-tuning and additional expert knowledge, which limits their scalability. In this work, we propose ReasoningGuard, an inference-time safeguard for LRMs. It injects timely safety aha moments during the reasoning process to guide the model towards harmless yet helpful reasoning. Our approach leverages the internal attention mechanisms of the LRM to accurately identify key points in the reasoning path, triggering safety-oriented reflections. To safeguard both the subsequent reasoning steps and the final answers, we implement a scaling sampling strategy during decoding to select the optimal reasoning path. With minimal additional inference cost, ReasoningGuard effectively mitigates four types of jailbreak attacks, including recent ones targeting the reasoning process of LRMs. Our approach outperforms nine existing safeguards, providing state-of-the-art defenses while avoiding common exaggerated safety issues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。