用自监督模型实现内窥镜手术场景的精准三维重建。
EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model
- 基于深度任意2的自监督框架，联合预测深度、位姿和镜头参数。
- 在SCARED和C3VD数据集上优于当前最优方法，重建精度显著提升。
- 适合做智能手术导航与增强现实系统的研发人员参考。
内窥镜手术场景的三维重建对提升场景感知、支持增强现实可视化及实现情境感知决策具有重要意义。其中，内窥镜相机内部参数的准确估计是关键但极具挑战的任务。真实手术环境中，无菌要求和可变焦、旋转镜头等特殊设备限制了传统标定方法的应用。现有多数重建方法不估计内部参数，导致重建可靠性受限。本文将内部参数估计融入自监督单目深度估计框架，通过适配深度任意2（DA2）模型实现深度、位姿与内参的联合预测。提出基于注意力的位姿网络和权重分解低秩微调（DoRA）策略，实现高效模型优化。在SCARED与C3VD公开数据集上验证，性能超越当前最新自监督单目深度估计与三维重建方法。代码与模型权重详见项目仓库：https://github.com/MOYF-beta/EndoSfM3D。
原文摘要 · Abstract (English)
3D reconstruction of endoscopic surgery scenes plays a vital role in enhancing scene perception, enabling AR visualization, and supporting context-aware decision-making in image-guided surgery. A critical yet challenging step in this process is the accurate estimation of the endoscope's intrinsic parameters. In real surgical settings, intrinsic calibration is hindered by sterility constraints and the use of specialized endoscopes with continuous zoom and telescope rotation. Most existing methods for endoscopic 3D reconstruction do not estimate intrinsic parameters, limiting their effectiveness for accurate and reliable reconstruction. In this paper, we integrate intrinsic parameter estimation into a self-supervised monocular depth estimation framework by adapting the Depth Anything V2 (DA2) model for joint depth, pose, and intrinsics prediction. We introduce an attention-based pose network and a Weight-Decomposed Low-Rank Adaptation (DoRA) strategy for efficient fine-tuning of DA2. Our method is validated on the SCARED and C3VD public datasets, demonstrating superior performance compared to recent state-of-the-art approaches in self-supervised monocular depth estimation and 3D reconstruction. Code and model weights can be found in project repository: https://github.com/MOYF-beta/EndoSfM3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。