让大模型信任自身安全直觉，无需人工标注即可自动防御攻击
Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense
- 利用模型内部信心差异生成自监督奖励信号
- 在15,000个无标签提示下实现超89%的防御成功率
- 适合追求低依赖、高鲁棒性的AI安全系统开发者
确保大语言模型（LLM）的安全性仍面临挑战，因缺乏通用标准和可靠的內容验证器，难以获得有效的训练信号。我们发现对齐模型已具备强大的内在安全信念：面对有害请求时始终产生高置信度拒绝，而在生成潜在危险内容时表现出高熵。这一熵差揭示了一个未被利用的信号——模型内在“知道”何时该拒绝。我们提出安全直觉强化学习（SIRL），将这种内部置信度转化为自生成奖励信号，摆脱对外部验证器或人工标注的依赖。SIRL通过强化低熵拒绝行为，教会模型信任其安全直觉。在Llama和Qwen模型上评估，SIRL在对抗20多种越狱方法（从静态提示到自适应攻击）时保持89%以上的防御成功率。仅使用1.5万条无标签提示，便超越资源密集的监督方法，同时保持数学、编码和对话基准性能。研究表明，有效对齐可源自模型内部，为无需大量人工干预的自主、鲁棒的AI安全机制铺平道路。
原文摘要 · Abstract (English)
Ensuring Large Language Model (LLM) safety remains challenging due to the absence of universal standards and reliable content validators, making it difficult to obtain effective training signals. We discover that aligned models already possess robust internal safety beliefs: they consistently produce high-confidence refusals to harmful requests while exhibiting high entropy when generating potentially dangerous content. This entropy gap reveals an untapped signal--models intrinsically "know" when to refuse. We introduce Safety Instincts Reinforcement Learning (SIRL), which transforms this internal confidence into a self-generated reward signal, eliminating dependence on external validators or human annotations. SIRL teaches models to trust their safety instincts by reinforcing low-entropy refusal behaviors. Evaluated on Llama and Qwen models, SIRL maintains 89%+ Defense Success Rates (DSRs) against 20+ jailbreak methods, from static prompts to adaptive attacks. Using only 15,000 unlabeled prompts, SIRL surpasses resource-intensive supervised methods while preserving performance on mathematics, coding, and conversation benchmarks. Our work demonstrates that effective alignment can emerge from within, paving the way for more autonomous and robust AI safety mechanisms that scale without extensive human oversight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。