arXiv:2507.04365cs.CRcs.AI2025-07被引 5

发现大模型越狱攻击中注意力逐渐偏移的机制,提出直接防御方法。

Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs

  • 揭示越狱攻击中注意力逐步削弱对危险请求响应的现象。
  • 四种主流大模型在多种攻击下均出现该现象,且防御效果与缓解程度正相关。
  • 提出注意力强化新防御,无需额外计算开销,适合实际部署。

随着大语言模型(LLMs)在社会与技术中的日益重要,其安全性愈发关键。越狱攻击利用漏洞绕过安全防护,构成重大威胁。然而,此类攻击的内在机制尚不清晰。本文揭示了一种普遍存在的现象:注意力滑移(Attention Slipping)。在攻击过程中,模型会逐步减少对用户查询中不安全请求的注意力,最终导致越狱成功。该现象在多种攻击方法中一致存在,包括基于梯度的令牌替换、提示模板优化和上下文学习。我们评估了两种基于查询扰动的防御方法——Token Highlighter 和 SmoothLLM,发现它们通过间接缓解注意力滑移起作用,且防御效果与缓解程度呈正相关。受此启发,我们提出注意力强化(Attention Sharpening),通过温度缩放直接增强注意力分布的尖锐性。在 Gema2-9B-It、Llama3.1-8B-It、Qwen2.5-7B-It、Mistral-7B-It v0.2 四个领先模型上实验表明,该方法能有效抵御各类越狱攻击,同时在 AlpacaEval 上保持良性任务性能。值得注意的是,注意力强化无额外计算或内存开销,是一种高效实用的现实部署方案。

原文摘要 · Abstract (English)

As large language models (LLMs) become more integral to society and technology, ensuring their safety becomes essential. Jailbreak attacks exploit vulnerabilities to bypass safety guardrails, posing a significant threat. However, the mechanisms enabling these attacks are not well understood. In this paper, we reveal a universal phenomenon that occurs during jailbreak attacks: Attention Slipping. During this phenomenon, the model gradually reduces the attention it allocates to unsafe requests in a user query during the attack process, ultimately causing a jailbreak. We show Attention Slipping is consistent across various jailbreak methods, including gradient-based token replacement, prompt-level template refinement, and in-context learning. Additionally, we evaluate two defenses based on query perturbation, Token Highlighter and SmoothLLM, and find they indirectly mitigate Attention Slipping, with their effectiveness positively correlated with the degree of mitigation achieved. Inspired by this finding, we propose Attention Sharpening, a new defense that directly counters Attention Slipping by sharpening the attention score distribution using temperature scaling. Experiments on four leading LLMs (Gemma2-9B-It, Llama3.1-8B-It, Qwen2.5-7B-It, Mistral-7B-It v0.2) show that our method effectively resists various jailbreak attacks while maintaining performance on benign tasks on AlpacaEval. Importantly, Attention Sharpening introduces no additional computational or memory overhead, making it an efficient and practical solution for real-world deployment.

越狱攻击注意力机制安全防御LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。