通过融合图像多区域信息，减少大模型幻觉问题。
MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs
- 利用跨注意力识别关键区域，分别生成答案。
- 基于JSD差异计算可靠性权重，融合区域预测结果。
- 无需训练，适合提升视觉语言模型的准确性。
大型视觉语言模型（LVLMs）在多模态任务中表现优异，但常因无法验证图像不同区域间的一致性而产生与视觉输入不符的幻觉。为此，我们提出无需训练的多区域融合解码方法（MRFD），通过建模区域间一致性来增强事实性。MRFD使用交叉注意力识别显著区域，对每个区域生成初始响应，并基于响应间的杰恩-申诺维特散度（JSD）计算可靠性权重。这些权重指导基于区域感知提示的、具有一致性意识的融合策略，灵感来自思维链推理。在多个LVLM和基准测试上的实验表明，该方法显著降低了幻觉率，提升了回答的事实性，且无需更新模型参数。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have shown strong performance across multimodal tasks. However, they often produce hallucinations -- text that is inconsistent with visual input, due to the limited ability to verify information in different regions of the image. To address this, we propose Multi-Region Fusion Decoding (MRFD), a training-free decoding method that improves factual grounding by modeling inter-region consistency. MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence (JSD) among the responses. These weights guide a consistency-aware fusion of per-region predictions, using region-aware prompts inspired by Chain-of-Thought reasoning. Experiments across multiple LVLMs and benchmarks show that MRFD significantly reduces hallucinations and improves response factuality without requiring model updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。