arXiv:2501.01834cs.CVcs.AI2025-01被引 8

用智能体协作让模型既懂专业又通常识,生成更准确的医学图像描述。

MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning

  • 拆解任务为问答链,专用模型专注看图,通用大模型负责提问和整合。
  • 在放射科报告生成上,生成内容准确性和流畅性显著提升。
  • 适合需要结合领域知识与通用理解的复杂图像描述场景。

图像字幕生成是计算机视觉与自然语言处理交叉的关键任务,广泛应用于多个领域。对于诊断报告生成等复杂任务,深度学习模型不仅需要领域特定的图像-字幕数据集,还需融合相关通用知识以保证上下文准确性。现有方法存在固有局限:专用模型擅长捕捉领域细节但泛化能力差,基于大语言模型(LLM)的视觉语言模型(VLM)虽具备通用知识,却难以适应特定领域。为此,本文提出一种新型代理增强型模型协作框架——MoColl,旨在有效融合领域特定知识与通用知识。具体而言,将复杂图像字幕任务分解为一系列相互关联的问答子任务。采用可训练的视觉问答(VQA)模型作为专用工具,基于图像内容回答任务相关问题;同时,基于LLM的代理利用通用知识提出问题,并将问答对合成连贯的字幕。除调用VQA模型外,该代理还通过反馈引导自身训练,提升领域适应能力。在放射科报告生成上的实验验证了所提框架的有效性,显著提升了生成报告的质量。

原文摘要 · Abstract (English)

Image captioning is a critical task at the intersection of computer vision and natural language processing, with wide-ranging applications across various domains. For complex tasks such as diagnostic report generation, deep learning models require not only domain-specific image-caption datasets but also the incorporation of relevant general knowledge to provide contextual accuracy. Existing approaches exhibit inherent limitations: specialized models excel in capturing domain-specific details but lack generalization, while vision-language models (VLMs) built on large language models (LLMs) leverage general knowledge but struggle with domain-specific adaptation. To address these limitations, this paper proposes a novel agent-enhanced model collaboration framework, which we call MoColl, designed to effectively integrate domain-specific and general knowledge. Specifically, our approach is to decompose complex image captioning tasks into a series of interconnected question-answer subtasks. A trainable visual question answering (VQA) model is employed as a specialized tool to focus on domain-specific visual analysis, answering task-specific questions based on image content. Concurrently, an LLM-based agent with general knowledge formulates these questions and synthesizes the resulting question-answer pairs into coherent captions. Beyond its role in leveraging the VQA model, the agent further guides its training to enhance its domain-specific capabilities. Experimental results on radiology report generation validate the effectiveness of the proposed framework, demonstrating significant improvements in the quality of generated reports.

图像字幕智能体协作医学影像多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。