医学视觉语言模型支持多模态图像与多种定位方式，提升医疗影像理解能力。
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine
- 融合语义分割与实例框定位，支持2D/3D医学图像
- 在多个任务上超越现有模型，尤其在标注任务中表现优异
- 适合医疗影像分析、智能诊断等临床研究者使用
近期视觉语言模型在生成视觉引导回答方面展现出巨大潜力，但在医疗领域仍面临独特挑战。例如，多数模型仅依赖单一视觉定位方法，而复杂医疗任务需要更灵活的处理方式；同时，多数模型仅处理2D图像，但大量医疗图像为3D数据；此外，医疗数据稀缺进一步加剧困难。为此，我们提出VividMed，一种支持多模态视觉定位的医学视觉语言模型。该模型可生成语义分割掩码和实例级边界框，兼容2D与3D成像模态。我们设计了三阶段训练流程，并基于公开数据集与模型构建自动数据合成管道。除视觉定位任务外，VividMed在视觉问答（VQA）与报告生成等下游任务中也表现出色。消融实验表明，引入视觉定位能力显著提升了各项任务性能。代码已开源：https://github.com/function2-llx/MMMM。
原文摘要 · Abstract (English)
Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable promise in generating visually grounded responses. However, their application in the medical domain is hindered by unique challenges. For instance, most VLMs rely on a single method of visual grounding, whereas complex medical tasks demand more versatile approaches. Additionally, while most VLMs process only 2D images, a large portion of medical images are 3D. The lack of medical data further compounds these obstacles. To address these challenges, we present VividMed, a vision language model with versatile visual grounding for medicine. Our model supports generating both semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. We design a three-stage training procedure and an automatic data synthesis pipeline based on open datasets and models. Besides visual grounding tasks, VividMed also excels in other common downstream tasks, including Visual Question Answering (VQA) and report generation. Ablation studies empirically show that the integration of visual grounding ability leads to improved performance on these tasks. Our code is publicly available at https://github.com/function2-llx/MMMM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。