用轻量模块让2D模型直接做3D/4D分割,参数只改不到4%。
SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting

- 通过深度路由LoRA和零成本特征交换模块,实现跨切片上下文建模。
- 仅微调2.8%~3.7%参数,在多个医学与生物数据集上表现领先。
- 适用于医疗影像与细胞追踪,支持从2D到3D/4D的无缝升级。
现有将2D基础模型(如SAM)适配至3D体积的方法要么独立处理切片(忽略切片间上下文),要么需大幅修改结构并重新训练。本文提出SAM+D,一种参数高效的框架,可将SAM系列模型提升一维空间:实现2D SAM到3D体积分割,首次通过参数高效微调实现基于视频的SAM2从2D+T到4D(3D+T)时空分割,同时冻结绝大部分预训练参数。SAM+D在冻结的Transformer块中引入两个轻量、模型无关模块:(1) 深度路由低秩适配(DRLoRA),通过学习路由实现空间自适应低秩更新;(2) 深度偏移模块(DSM),零额外参数成本实现跨切片特征交换。两者共同提供体素级上下文信息,仅需微调约2.8%(SAM)和3.7%(SAM2)参数。我们在两类场景验证:3D分割(在四个CT基准数据集KiTS、Pancreas、LiTS、Colon上评估)与4D分割(在细胞追踪挑战数据集CTC Fluo-N3DH-SIM+上评估)。在单点提示设置下,SAM+D均取得竞争力或更优结果,且参数量少于现有方法,证明其在不同架构、维度(3D、4D)及领域(医学影像、生物场景理解)间的泛化能力。代码已开源:https://github.com/JerrySongCST/SAM-Plus-D。
原文摘要 · Abstract (English)
Existing methods for adapting 2D foundation models such as SAM to 3D volumes either process slices independently---ignoring inter-slice context---or require substantial architectural changes and retraining. In this paper, we present \textbf{SAM+D}, a parameter-efficient framework that lifts SAM-family models by one spatial dimension---enabling 3D volumetric segmentation from 2D SAM and, for the first time via parameter-efficient fine-tuning, end-to-end 4D (3D+T) spatiotemporal segmentation from video-based SAM2---while keeping the vast majority of pre-trained parameters frozen. SAM+D introduces two lightweight, model-agnostic modules into frozen transformer blocks: (1)~\textbf{Depth-Routed LoRA (DRLoRA)} experts with learned routing for spatially adaptive low-rank updates, and (2)~\textbf{Depth Shift Modules (DSM)} for cross-slice feature exchange at zero additional parameter cost. Together, they provide volume-level context while tuning only ${\sim}$2.8\% of parameters for SAM and ${\sim}$3.7\% for SAM2. We evaluate SAM+D in two distinct settings, each lifting the base model by one spatial dimension: 3D segmentation, where SAM(2D$\,\to\,$3D) is evaluated on four CT benchmarks (KiTS, Pancreas, LiTS, Colon), and 4D segmentation, where SAM2 (2D+T$\,\to\,$3D+T) is evaluated on a cell tracking challenge (CTC) dataset (Fluo-N3DH-SIM+). In both settings SAM+D achieves competitive or superior results under the single-point prompt setting while using fewer trainable parameters than existing methods, demonstrating that SAM+D generalizes across SAM-family architectures, target dimensionalities (3D, 4D), and domains spanning medical imaging and bio-scene understanding. Code is publicly available at https://github.com/JerrySongCST/SAM-Plus-D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。