不训练即可抑制多模态推理中的幻觉,精准追踪问题相关的视觉线索。
ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoning
- 从问题出发,追踪线索在推理路径中的传播,定位关键视觉区域。
- 对所有推理模型提升1.21倍,非推理模型也提升1.14倍,无需训练。
- 无需修改模型,适配任意架构,适合研究和部署中减少幻觉的场景。
大型多模态推理模型通过显式的长链推理解决复杂视觉问题:从图像中提取视觉线索,并将其解码为文本标记。然而,这一能力也加剧了幻觉现象,即模型生成与输入图像或问题无关的内容。我们识别出‘推理漂移’:在线索收集过程中,模型过度关注与问题无关的实体,稀释了对任务相关线索的注意力,逐渐使推理过程脱离视觉依据。因此,许多针对非推理模型设计的定位或干预方法在推理场景中失效。基于此,我们提出ClueRecall评估指标,并引入ClueTracer——一种无需训练、无参数、架构无关的插件式幻觉抑制方法。ClueTracer从问题出发,沿推理路径(问题→输出→视觉标记)追踪关键线索,从而定位任务相关图像块,抑制对无关区域的错误关注。值得注意的是,该方法在不进行任何额外训练的情况下,使所有推理架构(包括R1-OneVision、Ocean-R1、MM-Eureka等)在推理基准上性能提升1.21倍;迁移至非推理设置时,性能提升1.14倍。
原文摘要 · Abstract (English)
Large multimodal reasoning models solve challenging visual problems via explicit long-chain inference: they gather visual clues from images and decode clues into textual tokens. Yet this capability also increases hallucinations, where the model generates content that is not supported by the input image or the question. To understand this failure mode, we identify \emph{reasoning drift}: during clue gathering, the model over-focuses on question-irrelevant entities, diluting focus on task-relevant cues and gradually decoupling the reasoning trace from visual grounding. As a consequence, many inference-time localization or intervention methods developed for non-reasoning models fail to pinpoint the true clues in reasoning settings. Motivated by these insights, we introduce ClueRecall, a metric for assessing visual clue retrieval, and present ClueTracer, a training-free, parameter-free, and architecture-agnostic plugin for hallucination suppression. ClueTracer starts from the question and traces how key clues propagate along the model's reasoning pathway (question $\rightarrow$ outputs $\rightarrow$ visual tokens), thereby localizing task-relevant patches while suppressing spurious attention to irrelevant regions. Remarkably, \textbf{without any additional training}, ClueTracer improves all \textbf{reasoning} architectures (including \texttt{R1-OneVision}, \texttt{Ocean-R1}, \texttt{MM-Eureka}, \emph{etc}.) by $\mathbf{1.21\times}$ on reasoning benchmarks. When transferred to \textbf{non-reasoning} settings, it yields a $\mathbf{1.14\times}$ gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。