提出无训练推理框架Fox，解决视觉语言模型幻觉问题
Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
- 用注意力熵探测定位导致幻觉的危险注意力头
- 通过逻辑值饱和干预切断幻觉路径，提升决策可信度
- 适合关注模型可解释性与生成真实性的研究者
大型视觉语言模型虽具复杂推理能力，却仍易产生对象幻觉。不同于主流关注注意力强度的观点，我们揭示更深层的结构错位：幻觉发生在决策关键步骤，特定注意力头作为高风险中介，脱离视觉证据而锁定语言先验，形成绕过视觉锚定的病理性捷径。为此，我们提出Fox（Faithfulness and Observational-flow via eXpression-rectification）——一种无需训练的推理时框架。Fox利用视觉注意力熵探针无监督定位危险中介；通过数值逻辑值饱和执行靶向因果干预，物理切断捷径路径；最后采用冲突门控协同解码策略，在保持生成流畅性的同时增强干预后的可信度。大量实验表明，Fox达到最优性能，相比SID提升29.1%且保留语言丰富性。代码已开源。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) exhibit sophisticated reasoning but remain susceptible to object hallucination. Deviating from the prevailing attention intensity assumption, we reveal a deeper dynamic structural misalignment: hallucination is triggered at decision-critical steps where specific attention heads, acting as risky mediators, decouple from visual evidence to lock onto language priors. This establishes a pathological shortcut that bypasses visual grounding. To dismantle this, we propose Fox (Faithfulness and Observational-flow via eXpression-rectification), a training-free inference-time framework. Fox diagnoses structural misalignment using a visual attention entropy probe to localize risky mediators unsupervisedly. We then execute a targeted causal intervention via numerical logit saturation to physically sever the shortcut path. Finally, a conflict-gated cooperative decoding strategy reconciles interventional faithfulness with observational fluency. Extensive experiments demonstrate that Fox achieves SOTA performance, outperforming SID by 29.1% while preserving linguistic richness. Code is available at https://github.com/Cc2021start/Fox.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。