让多模态大模型自主选择视觉区域并判断工具可靠性，提升推理准确性。
Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Framework
- 通过外部视觉模型与不确定性量化结合，指导模型自主选择关注区域。
- 在五个数据集上平均提升4.6%，部分表现超过微调方法。
- 无需训练，适合需要高可靠性的多模态推理场景。
多模态大语言模型（MLLM）在视觉问答等任务中展现潜力，但在多模态推理方面仍存挑战。现有方法采用代理框架或思维链（CoT）推理，但CoT需昂贵标注与微调，依赖外部工具的代理方法又可能引入不可靠输出。本文提出Seeing and Reasoning with Confidence（SRICE），一种无需训练的多模态推理框架，将外部视觉模型与不确定性量化（UQ）融入MLLM，通过多阶段交互实现模型自主选择感兴趣区域。我们采用基于置信区间预测的方法校准外部工具输出，并通过估计MLLM输出的不确定性来选择最优工具。实验表明，SRICE在五个数据集上相比基础MLLM平均提升4.6%，某些数据集性能甚至超越微调方法，验证了确保外部工具可靠使用对提升MLLM代理能力的重要性。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) show promise in tasks like visual question answering (VQA) but still face challenges in multimodal reasoning. Recent works adapt agentic frameworks or chain-of-thought (CoT) reasoning to improve performance. However, CoT-based multimodal reasoning often demands costly data annotation and fine-tuning, while agentic approaches relying on external tools risk introducing unreliable output from these tools. In this paper, we propose Seeing and Reasoning with Confidence (SRICE), a training-free multimodal reasoning framework that integrates external vision models with uncertainty quantification (UQ) into an MLLM to address these challenges. Specifically, SRICE guides the inference process by allowing MLLM to autonomously select regions of interest through multi-stage interactions with the help of external tools. We propose to use a conformal prediction-based approach to calibrate the output of external tools and select the optimal tool by estimating the uncertainty of an MLLM's output. Our experiment shows that the average improvement of SRICE over the base MLLM is 4.6% on five datasets and the performance on some datasets even outperforms fine-tuning-based methods, revealing the significance of ensuring reliable tool use in an MLLM agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。