通过感知潜在推理熵,减少多模态模型幻觉。
Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding
- 高熵时用概率加权连续嵌入,低熵时切回离散词元。
- 在多个基准上显著降低幻觉率,提升推理可靠性。
- 适合关注多模态推理准确性的研究者使用。
近期多模态大推理模型(MLRMs)在视觉问答任务中表现显著提升。然而我们发现,过渡词(如because、however、and wait)与幻觉密切相关,且常伴随高熵状态。我们认为,从令牌概率分布中可直接提取充分的上下文推理信息。受叠加表示理论启发,提出利用潜在叠加推理整合多重候选语义并保持推理轨迹。假设依赖离散文本输入会促使模型进行顺序显式推理,从而在高熵推理阶段忽视密集上下文线索。因此,我们构建基于令牌概率分布的丰富语义表征以增强上下文推理。为此,提出一种高效即插即用的解码策略——潜熵感知解码(LEAD),其核心为熵感知推理模式切换:高熵状态下采用概率加权连续嵌入,熵下降后返回离散词元嵌入。此外,提出先验引导的视觉锚点注入策略,促使模型聚焦视觉信息。大量实验表明,LEAD在多种MLRMs和多个基准上有效缓解幻觉。
原文摘要 · Abstract (English)
Recent advancements in multimodal large reasoning models (MLRMs) have significantly improved performance in visual question answering. However, we observe that transition words (e.g., because, however, and wait) are closely associated with hallucinations and tend to exhibit high-entropy states. We argue that adequate contextual reasoning information can be directly extracted from the token probability distribution. Inspired by superposed representation theory, we propose leveraging latent superposed reasoning to integrate multiple candidate semantics and maintain latent reasoning trajectories. The hypothesis is that reliance on discrete textual inputs may drive the model toward sequential explicit reasoning, underutilizing dense contextual cues during high-entropy reasoning stages. Therefore, we propose constructing rich semantic representations from the token probability distributions to enhance in-context reasoning. With this goal, we present Latent Entropy-Aware Decoding (LEAD), an efficient plug-and-play decoding strategy that leverages semantic context to achieve reliable reasoning. The heart of our method lies in entropy-aware reasoning mode switching. The model employs probability-weighted continuous embeddings under high-entropy states and transitions back to discrete token embeddings as entropy decreases. Moreover, we propose a prior-guided visual anchor injection strategy that encourages the model to focus on visual information. Extensive experiments show that LEAD effectively mitigates hallucinations across various MLRMs on multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。