医学影像分析新模型，兼顾3D结构与2D切片信息。
Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model
- 双通道编码：同时处理3D空间结构与2D切片内容。
- 任务导向注意力机制，提升关键切片识别准确率。
- 首个融合3D与2D特征的医学多模态大模型，适合临床研究者。
3D医学图像分析对现代医疗至关重要，但传统任务专用模型因泛化能力有限而难以应对多样临床场景。多模态大语言模型（MLLM）提供了潜在解决方案，但现有模型未能充分挖掘3D医学图像中丰富的层级信息。受临床实践启发——放射科医生既关注3D空间结构也重视2D切片内容——我们提出Med-2E3，一种融合双3D-2D编码器架构的3D医学MLLM。为高效聚合2D特征，设计了文本引导的跨切片（TG-IS）评分模块，根据切片内容和任务指令动态评估各切片注意力权重。据我们所知，Med-2E3是首个将3D与2D特征联合用于3D医学图像分析的MLLM。在大规模开源3D医学多模态数据集上的实验表明，TG-IS实现任务特异性注意力分布，显著优于当前最先进模型。代码已公开于https://github.com/MSIIP/Med-2E3。
原文摘要 · Abstract (English)
3D medical image analysis is essential for modern healthcare, yet traditional task-specific models are inadequate due to limited generalizability across diverse clinical scenarios. Multimodal large language models (MLLMs) offer a promising solution to these challenges. However, existing MLLMs have limitations in fully leveraging the rich, hierarchical information embedded in 3D medical images. Inspired by clinical practice, where radiologists focus on both 3D spatial structure and 2D planar content, we propose Med-2E3, a 3D medical MLLM that integrates a dual 3D-2D encoder architecture. To aggregate 2D features effectively, we design a Text-Guided Inter-Slice (TG-IS) scoring module, which scores the attention of each 2D slice based on slice contents and task instructions. To the best of our knowledge, Med-2E3 is the first MLLM to integrate both 3D and 2D features for 3D medical image analysis. Experiments on large-scale, open-source 3D medical multimodal datasets demonstrate that TG-IS exhibits task-specific attention distribution and significantly outperforms current state-of-the-art models. The code is available at: https://github.com/MSIIP/Med-2E3
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。