用梯度分析抑制多模态幻觉，让模型更听画面的话。
Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
- 通过一阶梯度分析文本与视觉特征的贡献，识别幻觉根源。
- 在多个评测中显著降低幻觉率，提升输出与图像的对齐度。
- 无需微调，可直接部署于现有模型，适合落地应用。
多模态大语言模型在各类任务中表现强劲，但依然容易产生幻觉，即输出与视觉输入不符。这一问题主要源于两种偏差：文本-视觉偏差（过度依赖提示和先前输出）和共现偏差（高频共现物体间的虚假关联）。本文提出基于梯度的影响感知约束解码（GACD），一种无需辅助模型、不需微调即可应用的推理阶段方法。核心是通过一阶泰勒梯度估计单个标记（视觉特征与文本标记）对当前输出的贡献。据此，GACD通过两个组件缓解幻觉：(1) 抑制与输出对象相关联的虚假视觉特征；(2) 通过增强视觉特征相对文本的权重来平衡跨模态贡献。在多个基准测试上的实验表明，GACD能有效减少幻觉，提升多模态大语言模型输出的视觉一致性。
原文摘要 · Abstract (English)
Multimodal large language models achieve strong performance across diverse tasks but remain prone to hallucinations, where outputs are not grounded in visual inputs. This issue can be attributed to two main biases: text-visual bias, the overreliance on prompts and prior outputs, and co-occurrence bias, spurious correlations between frequently paired objects. We propose Gradient-based Influence-Aware Constrained Decoding (GACD), an inference-based method, that addresses both biases without auxiliary models, and is readily applicable to existing models without finetuning. The core of our approach is bias estimation, which uses first-order Taylor gradients to understand the contribution of individual tokens-visual features and text tokens-to the current output. Based on this analysis, GACD mitigates hallucinations through two components: (1) suppressing spurious visual features correlated with the output objects, and (2) rebalancing cross-modal contributions by strengthening visual features relative to text. Experiments across multiple benchmarks demonstrate that GACD effectively reduces hallucinations and improves the visual grounding of MLLM outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。