arXiv:2606.32012cs.LGcs.CV2026-06被引 1

提出CoMet方法分解多模态模型不确定性,提升可信度判断

CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation

论文配图:CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
图 1 · 摘自论文原文
  • 将不确定性分解为上下文特异与答案多样性两部分
  • 在多个开放问答任务中优于现有基线,误差率降低12%-18%
  • 无需重复采样或自回归生成,推理效率高,适合部署

不确定性估计是人工智能模型长期面临的挑战,尤其在多模态大语言模型(MLLMs)中更为复杂。不确定性可能源于多种输入源及其交互关系,且在开放式回答场景下难以界定。为此,我们提出CoMet,一种通过分解不确定性为上下文特异性项和多重性特异性项的估计方法。前者捕捉由特定任务或提示引发的模糊性,后者量化在给定上下文下仍与输入兼容的合理答案数量。我们训练了一个轻量级后处理不确定性模块以估计这两项,实现高效无自回归生成的不确定性评估。在多个开放式多模态基准、幻觉检测及多项选择视觉问答任务上,CoMet均显著优于现有基线,同时保持实际应用中的高效性。代码已开源。

原文摘要 · Abstract (English)

Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf. the Dunning-Kruger effect). Although it is still far from solved even in simpler classification systems, tackling it in multimodal large language models (MLLMs) is becoming increasingly important. Within MLLMs, uncertainty can stem from any of the diverse sources as well as from their relationships, and further can stem from the unbounded answers in the open-ended setting. To tackle the issues, we propose CoMet, an MLLM uncertainty estimation method by decomposing uncertainty into a context-specific term and a multiplicity-specific term. The former captures ambiguity induced by the given context (e.g., task or prompt), while the latter captures how many plausible answers determined by the context remain compatible with the given input. We train a lightweight post-hoc uncertainty module to estimate these quantities, which enables efficient uncertainty estimation without autoregressive answer generation or repeated sampling. Experiments on various open-ended multimodal benchmarks, hallucination detection, and multiple-choice visual question answering benchmarks show that CoMet consistently improves uncertainty estimation over existing baselines while remaining efficient in practice. Code is available at https://github.com/princetonvisualai/comet_uncertainty

不确定性估计多模态模型LLMAI可信度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。