通过注意力黑洞诱导多模态大模型幻觉，仅需弱相关图文即可触发错误生成。
Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink
- 利用注意力机制漏洞，动态生成低相关性对抗图像触发幻觉。
- 在6个主流模型上成功攻击黑盒模型，包括GPT-4o与Gemini 1.5。
- 无需固定模式，攻击可迁移性强且不影响回答质量，适合安全评估。
将视觉理解融入语言生成的多模态大语言模型（MLLMs）正推动视觉-语言应用的发展。然而，这些模型常受幻觉问题困扰，即生成与视觉内容不符的物体、属性和关系。本文深入分析MLLMs内部注意力机制，揭示指令微调过程中的固有缺陷。提出一种新型幻觉攻击方法，利用注意力黑洞行为，在图像与文本相关性极低的情况下仍能触发幻觉内容，对关键下游应用构成重大威胁。与依赖固定模式的先前对抗方法不同，本方法生成动态、高效且高度可迁移的视觉对抗输入，同时保持模型输出质量。在6个主流MLLMs上的全面实验表明，该攻击即使面对广泛的缓解机制，仍能有效攻破黑盒模型，并在GPT-4o、Gemini 1.5等前沿商业API上取得显著效果。代码已公开于https://huggingface.co/RachelHGF/Mirage-in-the-Eyes。
原文摘要 · Abstract (English)
Fusing visual understanding into language generation, Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications. Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content. In this work, we delve into the internal attention mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing the inherent vulnerabilities in the instruction-tuning process. We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance, posing a significant threat to critical downstream applications. Distinguished from previous adversarial methods that rely on fixed patterns, our approach generates dynamic, effective, and highly transferable visual adversarial inputs, without sacrificing the quality of model responses. Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our attack in compromising black-box MLLMs even with extensive mitigating mechanisms, as well as the promising results against cutting-edge commercial APIs, such as GPT-4o and Gemini 1.5. Our code is available at https://huggingface.co/RachelHGF/Mirage-in-the-Eyes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。