通过因果建模双路径注意力，有效减少视觉语言模型的幻觉生成。
Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
- 基于因果图建模视觉与文本注意力的交互关系，识别幻觉成因。
- 提出VTACR指标，在低值场景下显著降低幻觉率。
- 双路径对比解码让真实视觉内容凸显，幻觉信息被抑制。
物体幻觉仍是大型视觉语言模型（LVLMs）的关键挑战，表现为生成内容与视觉输入不一致。现有基于语言解码器的方法通常独立调节视觉或文本注意力，忽视二者作为关键因果因素的交互作用。为此，我们提出Owl（Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation），一个基于因果框架的方法，通过结构因果图建模幻觉过程，将分解后的视觉和文本注意力视为中介变量。引入新指标VTACR（Visual-to-Textual Attention Contribution Ratio），量化解码过程中模态贡献的不平衡性。分析发现，幻觉多发生在低VTACR场景，此时文本先验主导，视觉锚定减弱。为此，我们设计细粒度注意力干预机制，根据VTACR信号动态调整逐标记、逐层注意力。最后，提出双路径对比解码策略：一条路径强化视觉驱动预测，另一条放大幻觉内容——使视觉真实性凸显，幻觉自然坍缩。在POPE和CHAIR基准测试中，Owl实现显著幻觉降低，保持视觉语言理解能力的同时达成忠实性新SOTA。代码已开源：https://github.com/CikZ2023/OWL。
原文摘要 · Abstract (English)
Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。