通过双指标引导对比对齐，提升多模态大模型的视觉推理可靠性。
DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models
- 用视觉注意力熵和图像输出相关性跟踪推理过程
- 异常时触发对比对齐，有效降低幻觉率
- 适合关注多模态模型可解释性与鲁棒性的研究者
人类视觉推理通常遵循从粗到细的注意力模式，先理解全局场景再聚焦问题相关区域。然而，多模态大语言模型可能因注意力漂移和视觉证据利用不足而偏离此模式，导致幻觉。为此，本文提出双指示器引导的对比对齐（DICA），在推理过程中追踪两个信息论指标：视觉注意力熵（VAE）反映视觉注意力集中程度，输出图像相关性（OIC）衡量生成结果对视觉输入的依赖程度。当VAE异常升高或OIC下降时，对应不同失效模式，触发针对性对比对齐以恢复视觉锚定。跨多个基准测试的实验表明，DICA持续优于现有方法，显著减少幻觉，验证了指标驱动干预在提升多模态推理可靠性方面的有效性。代码已公开于https://github.com/BGWH123/DICA/。
原文摘要 · Abstract (English)
Human visual reasoning typically follows a coarse-to-fine attention process, starting from global scene understanding and gradually focusing on question-relevant regions. However, multimodal large language models may deviate from this pattern due to attention drift and the underutilization of visual evidence, which can lead to hallucinations. To mitigate these issues, this study proposes a Dual-Indicator Guided Contrastive Alignment (DICA), which tracks two information-theoretic indicators during inference: Visual Attention Entropy (VAE), which reflects the concentration of visual attention, and Output Image Correlation (OIC), which measures the dependence of generated outputs on the visual input. An abnormal increase in VAE or a decrease in OIC corresponds to different failure modes, which trigger targeted contrastive alignment to restore visual grounding. Experimental results across multiple benchmarks demonstrate that DICA consistently outperforms existing approaches and substantially reduces hallucinations, highlighting the effectiveness of indicator-driven intervention in improving multimodal inference reliability. The code is publicly available at https://github.com/BGWH123/DICA/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。