用多模型动态加权解码，有效减少视觉语言模型幻觉
Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
- 推理时融合多个模型输出，按不确定性动态分配权重
- 在标准评测上显著降低幻觉率，且保持回答流畅性
- 无需训练，适合医疗、司法等高风险场景使用
大型视觉语言模型（LVLMs）在图像描述和视觉问答等任务中表现优异，但仍易产生物体幻觉——生成不存在或错误识别的物体描述。以往方法通过辅助训练目标或外部模块部分缓解此问题，但在可扩展性、适应性和模型独立性方面仍存在挑战。为此，我们提出训练无关的自适应标记集成解码（ATED），在推理阶段通过聚合多个LVLM的预测来减轻幻觉。ATED在每个解码步骤动态计算各模型的不确定性权重，反映其可靠性，并整合多样化解码路径以提升上下文关联与语义一致性。在标准幻觉检测基准上的实验表明，ATED显著优于现有方法，在不牺牲流畅性和相关性的情况下有效降低幻觉率。研究结果凸显了自适应集成的优势，为提升LVLM在高风险应用中的鲁棒性提供了新方向。代码已公开于https://github.com/jinlin2021/ATED。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have recently achieved impressive results in multimodal tasks such as image captioning and visual question answering. However, they remain prone to object hallucination -- generating descriptions of nonexistent or misidentified objects. Prior work has partially mitigated this via auxiliary training objectives or external modules, but challenges remain in terms of scalability, adaptability, and model independence. To address these limitations, we propose Adaptive Token Ensemble Decoding (ATED), a training-free, token-level ensemble framework that mitigates hallucination by aggregating predictions from multiple LVLMs during inference. ATED dynamically computes uncertainty-based weights for each model, reflecting their reliability at each decoding step. It also integrates diverse decoding paths to improve contextual grounding and semantic consistency. Experiments on standard hallucination detection benchmarks demonstrate that ATED significantly outperforms state-of-the-art methods, reducing hallucination without compromising fluency or relevance. Our findings highlight the benefits of adaptive ensembling and point to a promising direction for improving LVLM robustness in high-stakes applications. The code is available at https://github.com/jinlin2021/ATED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。