用分解卷积提升3D医学图像模型效率，性能超现有方法
DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions
- 将3D卷积拆分为三个1D卷积，降低计算开销
- 在5万张胸部CT上零样本检测18种病灶，效果领先
- 适合需要高效部署的临床3D多模态应用
视觉语言模型（VLMs）在2D医学图像分析中广泛应用，因其能对齐视觉与文本表征。但将其扩展至3D成像仍面临计算挑战。现有3D VLMs多依赖视觉变换器（ViTs），其自注意力机制具有二次复杂度；或采用3D卷积，随核尺寸增大导致参数量和浮点运算量激增。本文提出DCFormer，一种高效的3D图像编码器，通过将3D卷积分解为沿深度、高度和宽度方向的三个并行1D卷积，在保持空间信息的同时显著降低计算成本。该模型嵌入基于CLIP的视觉语言框架，在包含50,188对3D胸部CT与放射科报告的CT-RATE数据集上进行训练与评估。在18种病灶的零样本及微调检测任务，以及图像-文本检索任务中，DCFormer持续优于当前最优3D视觉编码器，包括CT-ViT、ViT、ConvNeXt、PoolFormer和TransUNet。结果表明，DCFormer具备构建可扩展、临床可用的3D医学视觉语言模型的潜力。代码已开源：https://github.com/mirthAI/DCFormer。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have been widely applied to 2D medical image analysis due to their ability to align visual and textual representations. However, extending VLMs to 3D imaging remains computationally challenging. Existing 3D VLMs often rely on Vision Transformers (ViTs), which are computationally expensive due to the quadratic complexity of self-attention, or on 3D convolutions, which require large numbers of parameters and FLOPs as kernel size increases. We introduce DCFormer, an efficient 3D image encoder that factorizes 3D convolutions into three parallel 1D convolutions along the depth, height, and width dimensions. This design preserves spatial information while significantly reducing computational cost. Integrated into a CLIP-based vision-language framework, DCFormer is trained and evaluated on CT-RATE, a dataset of 50,188 paired 3D chest CT volumes and radiology reports. In zero-shot and fine-tuned detection of 18 pathologies, as well as in image-text retrieval tasks, DCFormer consistently outperforms state-of-the-art 3D vision encoders, including CT-ViT, ViT, ConvNeXt, PoolFormer, and TransUNet. These results highlight DCFormer's potential for scalable, clinically deployable 3D medical VLMs. Our code is available at: https://github.com/mirthAI/DCFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。