用2D模型无训练生成3D医学图像嵌入，高效且性能领先。
Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
- 冻结2D预训练模型，从切片提取视觉特征并随机投影压缩
- 在10个任务上超越纯医学数据训练模型，最高提升14%
- 无需训练，适合快速部署于医疗影像分析场景
当前构建三维成像数据（如磁共振成像MRI）基础模型的挑战，源于高维空间中训练先进架构的计算复杂性以及大规模体积数据集的获取困难。为解决这些问题，我们提出Raptor（随机平面张量降维），一种无训练的三维医学体积嵌入生成方法。Raptor利用在自然图像上预训练的冻结2D基础模型，从医学体积的各个切片中提取视觉令牌，并通过随机投影进行空间压缩，显著降低计算复杂度的同时保留语义信息。在十个多样化的医学体积任务上的大量实验表明，Raptor在性能上优于现有最先进方法，包括仅在医学数据上预训练的模型（SuPreM提升+3%，MISFM提升+6%，Merlin提升+10%，VoCo提升+13%，SLIViT提升+14%），且完全避免了高昂的训练成本。结果凸显了Raptor作为推进基于深度学习的医学体积方法的基础的有效性与通用性。
原文摘要 · Abstract (English)
Current challenges in developing foundational models for volumetric imaging data, such as magnetic resonance imaging (MRI), stem from the computational complexity of training state-of-the-art architectures in high dimensions and curating sufficiently large datasets of volumes. To address these challenges, we introduce Raptor (Random Planar Tensor Reduction), a train-free method for generating semantically rich embeddings for volumetric data. Raptor leverages a frozen 2D foundation model, pretrained on natural images, to extract visual tokens from individual cross-sections of medical volumes. These tokens are then spatially compressed using random projections, significantly reducing computational complexity while retaining semantic information. Extensive experiments on ten diverse medical volume tasks verify the superior performance of Raptor over state-of-the-art methods, including those pretrained exclusively on medical volumes (+3% SuPreM, +6% MISFM, +10% Merlin, +13% VoCo, and +14% SLIViT), while entirely bypassing the need for costly training. Our results highlight the effectiveness and versatility of Raptor as a foundation for advancing deep learning-based methods for medical volumes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。