用多智能体协作提升超细节图像描述的准确性，解决大模型幻觉问题。
Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
- 设计多智能体系统，通过语言模型与视觉模型协作修正描述。
- 新评测框架更贴近人类对事实性的判断，现有方法在细节描述中效果有限。
- 发现VQA表现不等于生成详细描述的能力，适合关注真实性的研究者使用。
多模态大语言模型（MLLM）在生成高度细节化的图像描述方面表现出色，但常产生幻觉。我们的分析表明，现有幻觉检测方法在细节描述上表现不佳，原因是随着序列长度增加，MLLM越来越依赖自身生成文本而非输入图像。为此，我们提出一种多智能体方法，利用LLM与MLLM协作来纠正已有描述。同时，我们构建了评估框架和基准数据集，以系统化分析细节描述。实验显示，新评估方法比现有指标更贴近人类对事实性的判断；现有提升事实性的方法在超细节任务中可能失效。而我们的方法显著提升描述真实性，甚至改进了GPT-4V的输出。最后，我们指出以VQA为中心的评测存在局限性——其性能与生成详细描述能力无强相关性。代码与数据已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MLLM factuality may fall short in hyper-detailed image captioning tasks. In contrast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions. Our code and data are available at https://github.com/adobe-research/CapMAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。