通过原子级可信度评估，有效减少多模态大模型的幻觉问题。
Taming Object Hallucinations with Verified Atomic Confidence Estimation
- 将回答分解为原子查询，用重述降低表述敏感性。
- 在5个基准上显著降低幻觉率，提升可信度校准效果。
- 无需外部专家，适合提升视觉语言模型的可靠性。
多模态大语言模型（MLLMs）常出现物体存在、属性或关系等方面的幻觉，影响其可靠性。本文提出TACO（验证性原子可信度估计）框架，通过自我验证与可信度校准，不依赖外部视觉专家即可缓解幻觉。TACO将输出分解为原子级查询，对查询进行重述以降低对措辞的敏感性，并采用自一致性（黑箱）或自信心（灰箱）聚合方式估计可信度，再由语言模型精炼答案。在五个基准（POPE、MME、HallusionBench、AMBER、MM-Hal Bench）上，使用两种MLLM（LLaVA-1.5-7B和CogVLM2）的实验表明，TACO持续优于直接提示和视觉对比解码，减少系统性偏差并改善可信度校准，证明其在提升MLLM忠实性方面的有效性。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) often suffer from hallucinations, particularly errors in object existence, attributes, or relations, which undermine their reliability. We introduce TACO (Verified Atomic Confidence Estimation), a simple framework that mitigates hallucinations through self-verification and confidence calibration without relying on external vision experts. TACO decomposes responses into atomic queries, paraphrases them to reduce sensitivity to wording, and estimates confidence using self-consistency (black-box) or self-confidence (gray-box) aggregation, before refining answers with a language model. Experiments on five benchmarks (POPE, MME, HallusionBench, AMBER, and MM-Hal Bench) with two MLLMs (\texttt{LLaVA-1.5-7B} and \texttt{CogVLM2}) show that TACO consistently outperforms direct prompting and Visual Contrastive Decoding, reduces systematic biases, and improves confidence calibration, demonstrating its effectiveness in enhancing the faithfulness of MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。