通过动态文本扰动，缓解多模态模型的幻觉问题。
Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
- 在解码阶段对文本进行分层扰动，识别并抑制语言先验干扰。
- 在多个基准上显著降低幻觉率，提升视觉接地准确性。
- 无需训练，适合希望提升生成可靠性的开发者使用。
多模态大语言模型常因语言先验过度主导视觉证据而产生推理幻觉。现有无训练缓解方法或扰动视觉表征导致偏离自然图像分布，或引入侵入性操作损害生成流畅性。本文提出新视角：多模态幻觉表现为解码阶段视觉定位对文本表述的过度敏感。基于此，我们提出训练无关的Decoding by Perturbation（DeP）框架，通过动态探针施加多层级文本扰动，激发潜在语言先验。利用注意力方差增强稳定特征区域，抑制噪声；同时基于逻辑值统计构建可解释的先验漂移方向，抵消文本共现带来的概率偏差。大量实验表明，DeP有效减少幻觉，在多个基准上表现优异。
原文摘要 · Abstract (English)
Multimodal Large Language Models frequently suffer from inference hallucinations, partially stemming from language priors dominating visual evidence. Existing training-free mitigation methods either perturb the visual representation and deviate from the natural image distribution, or enforce intrusive manipulations that compromise the model's inherent generative fluency. We introduce a novel perspective that multimodal hallucination manifests as the hypersensitivity of visual grounding to textual phrasing during the decoding phase. Building on this insight, we propose Decoding by Perturbation (DeP), a training-free framework mitigating prior-induced hallucinations via controlled textual interventions. DeP employs a dynamic probe applying multi-level textual perturbations to elicit latent language priors. Leveraging attention variance, it enhances stable evidence regions while suppressing suspicious noise in the feature space. Furthermore, it constructs an interpretable prior drift direction using logits statistics to counteract probability biases from textual co-occurrences. Extensive experiments confirm DeP effectively reduces hallucinations and achieves superior performance across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。