arXiv:2602.09541cs.CV2026-02中稿 · the first round, w…

用高斯混合桥精准调整注意力分布,减少多模态幻觉。

Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucination

  • 通过高斯混合模型建模可信与幻觉的注意力分布
  • 利用熵最优传输实现注意力成分的精确映射
  • 无需额外计算,适合各类视觉语言模型部署

大型视觉语言模型在多模态任务中表现优异,但因语言模型先验过强及模态间注意力错位,常生成与图像内容不符的幻觉输出。为此,本文提出 Scalpel,通过细化注意力激活分布来缓解幻觉。该方法在推理阶段预测每个 Transformer 头的可信注意力方向,并动态调整其激活值。Scalpel 利用高斯混合模型捕捉可信与幻觉注意力流形中的多峰分布,通过熵最优传输(等价于 Schrödinger 桥问题)实现对高斯成分的精确映射。在干预过程中,根据成分归属和幻觉-可信激活间的映射关系,动态调节干预强度与方向。在多个数据集与基准上的实验表明,Scalpel 有效抑制幻觉,性能优于现有方法,达到当前最佳水平。此外,Scalpel 具备模型与数据无关性,仅需一次解码,不增加额外计算开销。

原文摘要 · Abstract (English)

Rapid progress in large vision-language models (LVLMs) has achieved unprecedented performance in vision-language tasks. However, due to the strong prior of large language models (LLMs) and misaligned attention across modalities, LVLMs often generate outputs inconsistent with visual content - termed hallucination. To address this, we propose \textbf{Scalpel}, a method that reduces hallucination by refining attention activation distributions toward more credible regions. Scalpel predicts trusted attention directions for each head in Transformer layers during inference and adjusts activations accordingly. It employs a Gaussian mixture model to capture multi-peak distributions of attention in trust and hallucination manifolds, and uses entropic optimal transport (equivalent to Schrödinger bridge problem) to map Gaussian components precisely. During mitigation, Scalpel dynamically adjusts intervention strength and direction based on component membership and mapping relationships between hallucination and trust activations. Extensive experiments across multiple datasets and benchmarks demonstrate that Scalpel effectively mitigates hallucinations, outperforming previous methods and achieving state-of-the-art performance. Moreover, Scalpel is model- and data-agnostic, requiring no additional computation, only a single decoding step.

多模态幻觉抑制注意力机制生成质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。