arXiv:2512.03542cs.CVcs.AI2025-12被引 3

通过视觉推理时干预,有效减少多模态大模型的幻觉问题。

V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention

  • 基于注意力头激活模式检测视觉忽视,决定何时干预。
  • 仅在发现视觉忽视时调用预存视觉特征,避免过度干预。
  • 在8个基准上提升视觉一致性,不降低通用任务性能。

多模态大语言模型在视觉-语言任务中表现优异,但存在与输入图像不符的幻觉问题,影响高精度场景的可靠性。根本原因在于视觉忽视——模型未能充分关注输入图像。现有方法多通过修改注意力分数或输出逻辑来缓解,却忽视了“何时干预”这一前提,导致过度干预、引入新幻觉及计算开销。本文首先分析视觉忽视机制,发现可通过多模态大模型的头级激活模式精准检测。为此提出V-ITI:一个轻量级推理时干预框架,包含视觉忽视检测器(基于头级判别探针)和视觉召回干预器(仅在检测到忽视时,用预存视觉激活信息调制模型)。在8个不同架构的多模态大模型基准上验证,V-ITI持续降低视觉相关幻觉,同时保持原有任务性能。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations, producing content inconsistent with input visuals, that undermine reliability in precision-sensitive domains. This issue stems from a fundamental problem of visual neglect, where models fail to adequately prioritize input images. Existing methods typically alleviate hallucinations by intervening in the attention score or output logits, focusing on "how to intervene" but overlooking the prerequisite "when to intervene", which leads to the "over-intervention" problem and subsequently introduces new hallucinations and unnecessary computational overhead. To address this gap, we first investigate the mechanism of visual neglect and reveal it can be accurately detected via head-level activation patterns in MLLMs. We thus propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector that identifies visual neglect via head-level discriminative probes and a Visual Recall Intervenor that modulates activations with prestored visual activation information only when the visual neglect is detected. Extensive experiments across eight benchmarks and different MLLM families demonstrate that V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.

多模态幻觉抑制推理干预视觉忽视

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。