用快速傅里叶方法高效微调大模型,提升腹腔镜肝部关键点分割精度。
Subsampled Randomized Fourier GaLore for Adapting Foundation Models in Depth-Driven Liver Landmark Segmentation
- 用随机子采样傅里叶变换替代SVD,低成本微调大模型注意力层。
- 在L3D数据集上Dice提升4.85%,表面距离降低11.78点。
- 适用于实时腹腔镜手术场景,跨数据集泛化能力强。
准确检测与勾画医学影像中的解剖结构对计算机辅助手术至关重要,尤其在腹腔镜肝脏手术中,二维视频流限制了深度感知,增加了关键点定位难度。尽管近期工作利用单目深度线索提升了关键点检测效果,但在融合RGB与深度特征以及高效适配大规模视觉模型方面仍存在挑战。本文提出一种深度引导的肝部关键点分割框架,通过视觉基础编码器融合语义与几何信息。采用Segment Anything Model V2(SAM2)提取RGB特征,Depth Anything V2(DA2)提取深度感知特征。为高效微调SAM2,提出SRFT-GaLore方法,以子采样随机傅里叶变换(SRFT)替代计算量大的SVD,实现高维注意力层的高效微调且不损失表征能力。引入交叉注意力融合模块整合多模态线索。为评估跨数据集泛化能力,构建新公开的腹腔镜肝脏手术数据集LLSD作为外部验证基准。在公开的L3D数据集上,本方法相较D2GPLand提升4.85% Dice相似系数,平均对称表面距离降低11.78点。进一步在LLSD上评估,模型保持竞争力并显著优于基于SAM的基线,展现出强跨数据集鲁棒性与对未知手术环境的适应能力。结果表明,该SRFT-GaLore增强的双编码器框架可在实时、深度受限的外科环境下实现可扩展、精准的分割。
原文摘要 · Abstract (English)
Accurate detection and delineation of anatomical structures in medical imaging are critical for computer-assisted interventions, particularly in laparoscopic liver surgery where 2D video streams limit depth perception and complicate landmark localization. While recent works have leveraged monocular depth cues for enhanced landmark detection, challenges remain in fusing RGB and depth features and in efficiently adapting large-scale vision models to surgical domains. We propose a depth-guided liver landmark segmentation framework integrating semantic and geometric cues via vision foundation encoders. We employ Segment Anything Model V2 (SAM2) encoder to extract RGB features and Depth Anything V2 (DA2) encoder to extract depth-aware features. To efficiently adapt SAM2, we introduce SRFT-GaLore, a novel low-rank gradient projection method that replaces the computationally expensive SVD with a Subsampled Randomized Fourier Transform (SRFT). This enables efficient fine-tuning of high-dimensional attention layers without sacrificing representational power. A cross-attention fusion module further integrates RGB and depth cues. To assess cross-dataset generalization, we also construct a new Laparoscopic Liver Surgical Dataset (LLSD) as an external validation benchmark. On the public L3D dataset, our method achieves a 4.85% improvement in Dice Similarity Coefficient and a 11.78-point reduction in Average Symmetric Surface Distance compared to the D2GPLand. To further assess generalization capability, we evaluate our model on LLSD dataset. Our model maintains competitive performance and significantly outperforms SAM-based baselines, demonstrating strong cross-dataset robustness and adaptability to unseen surgical environments. These results demonstrate that our SRFT-GaLore-enhanced dual-encoder framework enables scalable and precise segmentation under real-time, depth-constrained surgical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。