arXiv:2503.20047cs.CVeess.IV2025-03被引 65

高效融合3D医学影像与文本的视觉语言模型,提升多任务临床分析能力。

Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis

  • 采用分解3D卷积的DCFormer,高效捕捉大尺度空间特征。
  • 在图像文本检索中达61.00% R@1,报告生成METEOR达36.42%。
  • 适合需要3D影像多模态理解的临床研究与辅助诊断场景。

视觉语言模型(VLMs)在二维医学图像分析中展现潜力,但将其扩展至三维仍面临体积数据高计算需求及三维空间特征与临床文本对齐困难的问题。我们提出Med3DVLM,一种针对这些挑战的3D VLM,包含三项关键创新:(1) DCFormer,一种利用分解3D卷积的高效编码器,可大规模捕获细粒度空间特征;(2) SigLIP,一种基于成对sigmoid损失的对比学习策略,无需依赖大规模负样本即可提升图像-文本对齐;(3) 双流MLP-Mixer投影器,融合低层与高层图像特征与文本嵌入,生成更丰富的多模态表示。我们在包含120,084张3D医学影像及放射科报告、VQA数据的M3D数据集上评估该模型。结果显示,Med3DVLM在多个基准测试中表现优异:图像-文本检索达到61.00% R@1(优于当前SOTA模型的19.10%);报告生成获得36.42% METEOR得分(高于14.38%);开闭式VQA分别达36.76% METEOR(vs. 33.58%)和79.95%准确率(vs. 75.78%)。结果表明,Med3DVLM能有效连接3D影像与语言,支持可扩展的多任务临床推理。代码已公开于https://github.com/mirthAI/Med3DVLM。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have shown promise in 2D medical image analysis, but extending them to 3D remains challenging due to the high computational demands of volumetric data and the difficulty of aligning 3D spatial features with clinical text. We present Med3DVLM, a 3D VLM designed to address these challenges through three key innovations: (1) DCFormer, an efficient encoder that uses decomposed 3D convolutions to capture fine-grained spatial features at scale; (2) SigLIP, a contrastive learning strategy with pairwise sigmoid loss that improves image-text alignment without relying on large negative batches; and (3) a dual-stream MLP-Mixer projector that fuses low- and high-level image features with text embeddings for richer multi-modal representations. We evaluate our model on the M3D dataset, which includes radiology reports and VQA data for 120,084 3D medical images. Results show that Med3DVLM achieves superior performance across multiple benchmarks. For image-text retrieval, it reaches 61.00% R@1 on 2,000 samples, significantly outperforming the current state-of-the-art M3D model (19.10%). For report generation, it achieves a METEOR score of 36.42% (vs. 14.38%). In open-ended visual question answering (VQA), it scores 36.76% METEOR (vs. 33.58%), and in closed-ended VQA, it achieves 79.95% accuracy (vs. 75.78%). These results highlight Med3DVLM's ability to bridge the gap between 3D imaging and language, enabling scalable, multi-task reasoning across clinical applications. Our code is publicly available at https://github.com/mirthAI/Med3DVLM.

3D医学影像视觉语言模型多模态临床应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。