让视觉语言模型不再胡编乱造,用背景一致的方式隐藏物体
Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models
- 通过重新编码物体视觉特征使其与背景一致来隐藏目标
- 可保留86%非目标物体,幻觉减少3倍以上
- 适合研究模型鲁棒性或对抗攻击的开发者
视觉语言模型在视觉理解与生成方面表现卓越,但易受视觉内容的对抗性干扰。以往的物体隐藏攻击多依赖抑制或屏蔽特定区域的表征,常造成语义断层,引发模型虚构出合理却错误的物体(即幻觉)。本文指出,幻觉并非源于物体缺失本身,而是由抑制类攻击引入的语义不连续所致。为此,提出一类新的‘背景一致性物体隐藏’攻击方法:将目标物体的视觉表征重新编码为与周围背景在统计和语义上一致,同时保持标记结构与注意力流,避免产生表征空洞。设计了像素级优化框架,在多个Transformer层中实现背景一致性重编码,并维持全局场景语义。在多个先进视觉语言模型上的实验表明,该方法能有效隐藏目标物体,同时保留高达86%的非目标物体,相比基于注意力抑制的攻击,接地幻觉减少达3倍。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have recently shown remarkable capabilities in visual understanding and generation, but remain vulnerable to adversarial manipulations of visual content. Prior object-hiding attacks primarily rely on suppressing or blocking region-specific representations, often creating semantic gaps that inadvertently induce hallucination, where models invent plausible but incorrect objects. In this work, we demonstrate that hallucination arises not from object absence per se, but from semantic discontinuity introduced by such suppression-based attacks. We propose a new class of \emph{background-consistent object concealment} attacks, which hide target objects by re-encoding their visual representations to be statistically and semantically consistent with surrounding background regions. Crucially, our approach preserves token structure and attention flow, avoiding representational voids that trigger hallucination. We present a pixel-level optimization framework that enforces background-consistent re-encoding across multiple transformer layers while preserving global scene semantics. Extensive experiments on state-of-the-art vision-language models show that our method effectively conceals target objects while preserving up to $86\%$ of non-target objects and reducing grounded hallucination by up to $3\times$ compared to attention-suppression-based attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。