构建3D医学影像推理数据集,提升大模型空间理解能力
Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models

- 通过切片级合成构建结构化推理数据,模拟放射科医生诊断流程
- 在多个3D医学基准上显著超越2D基线模型,接近专用3D架构性能
- 无需3D预训练即可实现可解释的立体空间推理,适合临床辅助系统
尽管多模态大语言模型在2D医学图像理解中表现优异,但其向3D体数据扩展仍受限于高昂标注成本和数据不透明。现有数据格式多为固定的视觉问答对或非结构化临床报告,难以捕捉明确的临床推理过程。为此,我们提出一种新颖的切片级数据合成范式,构建大规模结构化推理数据集。该范式受放射科医生真实诊断流程启发,将复杂的3D阅读过程分解为细粒度的逐切片观察,并合成可解释的思维链(Chain-of-Thought)。关键在于,该框架强制执行重要临床原则:顺序空间追踪、多切片空间感知以抑制伪影,以及差异性排除。为验证方法有效性,我们使用合成数据对标准2D预训练MLLM进行指令微调,以增强其体积理解能力。在多个3D医学基准上的全面评估表明,该方法显著优于2D基线模型,且生成模型展现出鲁棒的空间推理能力,媲美计算成本高昂的原生3D架构,有效弥合性能差距。最终,这一数据驱动策略在无需3D特定预训练的前提下,实现了深层体积理解与高度可解释的临床逻辑。完整代码库(含数据集与训练流程)已公开于 https://github.com/2020420145009/hounsfield。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。