arXiv:2605.01766cs.LGcs.CV2026-05被引 1

不训练模型,推理时增强视觉音频输入的使用,减少幻觉。

Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time

论文配图:Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
图 1 · 摘自论文原文
  • 用层间相关性分析量化各模态贡献,推理时动态强化感知输入
  • 在多个视觉与音频基准上显著降低幻觉率,保持生成质量
  • 无需额外训练,适合追求高可靠性的多模态应用

多模态大语言模型(MLLMs)在复杂视觉与语音语言任务中展现出强大能力,但其常出现幻觉,即输出偏离实际感知输入。这源于推理时文本令牌主导,导致模型过度依赖语言先验而非真实输入。为此,我们提出训练无关的推理时模态增强框架LIME,利用层间相关性传播(LRP)量化每个词元的贡献,构建基于相关性的目标函数,通过调整解码过程中的键值表示来增强感知输入的使用,无需修改模型参数或新增训练数据。我们在多个视觉与音频领域的多模态基准上评估该方法,结果表明其能持续降低幻觉率并提升模型对输入的依从性,同时保持生成质量。进一步分析显示,LIME提高了模态贡献度,并生成更局部、语义对齐的相关性分布。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) have revolutionized the landscape of AI, demonstrating impressive capabilities in tackling complex vision and audio-language tasks. However, a critical challenge remains: these models often suffer from hallucinations, generating outputs that diverge from the provided perceptual inputs. This tendency stems from an inherent imbalance in modality utilization during inference, where the dominance of textual tokens undermines the potential of perceptual inputs. As a result, the model frequently resorts to textual language priors at the expense of grounded evidence. To tackle this issue, we propose Learning Inference-time Modality Enhancement (LIME), a training-free framework designed to bolster multimodal grounding by explicitly enhancing modality usage during decoding. LIME leverages Layer-wise Relevance Propagation (LRP) to quantify token-level contributions and defines a relevance-based objective that promotes increased reliance on perceptual inputs. This objective is enforced through inference-time updates to the model's key-value representations, without modifying model parameters or requiring additional training data. We evaluate LIME across multiple multimodal benchmarks in both vision and audio domains, demonstrating consistent reductions in hallucinations and enhanced grounding while preserving generation quality. Further analysis shows that LIME increases modality contribution and produces more localized and semantically aligned relevance patterns.

多模态幻觉抑制推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。