用共享特征统一表示内窥镜图像与深度,提升单目深度估计精度。
MetaFE-DE: Learning Meta Feature Embedding for Depth Estimation from Monocular Endoscopic Images
- 提出元特征嵌入(MetaFE),用同一组特征同时还原RGB和深度图
- 两阶段自监督学习:先用扩散模型建模时空特征,再解码生成深度图
- 在多个数据集上超越现有方法,适合临床内窥镜辅助系统研发
单目内窥镜图像的深度估计因手术场景复杂、组织形状不规则及光照变化大而极具挑战。现有方法多直接从RGB图像推断深度,常面临可解释性差、精度不足的问题。本文提出“元特征嵌入”(MetaFE)概念,将手术中的物理实体(如组织、器械)以可双向解码为RGB或深度图的共享特征表示。基于此,设计两阶段自监督学习框架:第一阶段利用扩散模型构建时序表征,并通过交叉归一化对齐空间信息,形成MetaFE;第二阶段采用带亮度校准的自监督单目深度估计,解码元特征生成深度图。在多个内窥镜数据集上的实验证明,本方法显著优于现有最先进方法,在精度和泛化能力上均有提升。源代码将公开。
原文摘要 · Abstract (English)
Depth estimation from monocular endoscopic images presents significant challenges due to the complexity of endoscopic surgery, such as irregular shapes of human soft tissues, as well as variations in lighting conditions. Existing methods primarily estimate the depth information from RGB images directly, and often surffer the limited interpretability and accuracy. Given that RGB and depth images are two views of the same endoscopic surgery scene, in this paper, we introduce a novel concept referred as ``meta feature embedding (MetaFE)", in which the physical entities (e.g., tissues and surgical instruments) of endoscopic surgery are represented using the shared features that can be alternatively decoded into RGB or depth image. With this concept, we propose a two-stage self-supervised learning paradigm for the monocular endoscopic depth estimation. In the first stage, we propose a temporal representation learner using diffusion models, which are aligned with the spatial information through the cross normalization to construct the MetaFE. In the second stage, self-supervised monocular depth estimation with the brightness calibration is applied to decode the meta features into the depth image. Extensive evaluation on diverse endoscopic datasets demonstrates that our approach outperforms the state-of-the-art method in depth estimation, achieving superior accuracy and generalization. The source code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。