arXiv:2508.05205cs.CV2025-08

通过多领域预训练,实现内窥镜图像的强泛化匹配。

EndoMatcher: Generalizable Endoscopic Image Matcher via Multi-Domain Pre-training for Robot-Assisted Surgery

  • 双分支Vision Transformer+交互模块,增强弱纹理下的特征匹配能力。
  • 在6个领域120万张图像上训练,零样本下匹配精度提升超200%。
  • 适合机器人手术中3D重建、导航等需跨器官泛化的场景。

内窥镜图像的泛化稠密特征匹配对机器人辅助手术中的三维重建、导航和术野理解至关重要,但受限于弱纹理、视角变化大及标注数据稀缺等挑战。为此,本文提出EndoMatcher,一种基于大规模多领域数据预训练的通用内窥镜图像匹配方法。为应对复杂视觉条件,采用双分支Vision Transformer提取多尺度特征,并引入双重交互模块强化对应关系学习。为解决数据稀缺与域多样性问题,构建首个多领域内窥镜匹配数据集Endo-Mix6,包含约120万张真实与合成图像对,覆盖六个不同领域,对应关系标签通过SfM与模拟变换生成。该数据集规模与分布差异带来训练不稳定性。为此,设计渐进式多目标训练策略,促进跨域平衡学习,提升表示质量。实验表明,EndoMatcher在未见器官和成像条件下可实现零样本泛化,在Hamlyn和Bladder数据集上内点匹配数分别提升140.69%和201.43%,在Gastro-Matching数据集上匹配方向预测准确率(MDPA)提升9.40%,在复杂内窥镜条件下实现高密度且精准的匹配。代码已开源。

原文摘要 · Abstract (English)

Generalizable dense feature matching in endoscopic images is crucial for robot-assisted tasks, including 3D reconstruction, navigation, and surgical scene understanding. Yet, it remains a challenge due to difficult visual conditions (e.g., weak textures, large viewpoint variations) and a scarcity of annotated data. To address these challenges, we propose EndoMatcher, a generalizable endoscopic image matcher via large-scale, multi-domain data pre-training. To address difficult visual conditions, EndoMatcher employs a two-branch Vision Transformer to extract multi-scale features, enhanced by dual interaction blocks for robust correspondence learning. To overcome data scarcity and improve domain diversity, we construct Endo-Mix6, the first multi-domain dataset for endoscopic matching. Endo-Mix6 consists of approximately 1.2M real and synthetic image pairs across six domains, with correspondence labels generated using Structure-from-Motion and simulated transformations. The diversity and scale of Endo-Mix6 introduce new challenges in training stability due to significant variations in dataset sizes, distribution shifts, and error imbalance. To address them, a progressive multi-objective training strategy is employed to promote balanced learning and improve representation quality across domains. This enables EndoMatcher to generalize across unseen organs and imaging conditions in a zero-shot fashion. Extensive zero-shot matching experiments demonstrate that EndoMatcher increases the number of inlier matches by 140.69% and 201.43% on the Hamlyn and Bladder datasets over state-of-the-art methods, respectively, and improves the Matching Direction Prediction Accuracy (MDPA) by 9.40% on the Gastro-Matching dataset, achieving dense and accurate matching under challenging endoscopic conditions. The code is publicly available at https://github.com/Beryl2000/EndoMatcher.

医学图像特征匹配机器人手术多域学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。