用高光谱成像提升微创手术中关键点检测精度。
HyKey: Hyperspectral Keypoint Detection and Matching in Minimally Invasive Surgery
- 融合空间与光谱特征的3D-2D混合网络，提升弱纹理环境下的关键点提取能力。
- 在真实手术场景下实现96.62%匹配准确率和67.18%姿态估计准确率。
- 适合需要高鲁棒性3D重建的微创手术视觉系统研发者。
目的：微创手术中的3D重建可通过增强可视化、器械追踪和增强现实来提升手术引导效果。然而，传统基于RGB的关键点检测与匹配方法在缺乏纹理和光照复杂等手术环境下表现不佳。本文探究了快照式高光谱成像（HSI）是否能改善手术场景中的关键点检测与匹配性能。方法：我们提出HyKey模型，采用混合3D-2D卷积神经网络联合提取高光谱图像的空间-光谱特征。模型在机器人采集的双相机RGB-HSI腹腔镜数据集上训练，该数据集包含离体器官且相机位姿已校准，采用合成同胚变换和对极几何约束进行数据增强。与SuperPoint、ALIKE等主流RGB方法对比，结果表明，基于HSI的模型在注册后的RGB帧上表现更优，达到96.62%的平均匹配准确率和10度姿态估计下67.18%的平均精度。结论：将高光谱立方体中的光谱信息融入，为微创手术中鲁棒的单目3D重建提供了新路径，有效缓解低纹理环境带来的挑战。代码与数据集已开源。
原文摘要 · Abstract (English)
Purpose: 3D reconstruction in minimally invasive surgery (MIS) enables enhanced surgical guidance through improved visualisation, tool tracking, and augmented reality. However, traditional RGB-based keypoint detection and matching pipelines struggle with surgical challenges, such as poor texture and complex illumination. We investigate whether using snapshot hyperspectral imaging (HSI) can provide improved results on keypoint detection and matching surgical scenes. Methods: We developed HyKey, a HYperspectral KEYpoint detection and description model made up of a hybrid 3D-2D convolutional neural network that jointly extracts spatial-spectral features from HSI. The model was trained using synthetic homographic augmentation and epipolar geometry constraints on a robotically-acquired dual-camera RGB-HSI laparoscopic dataset of ex-vivo organs with calibrated camera poses. We benchmarked performance against established RGB-based methods, including SuperPoint and ALIKE. Results: Our HSI-based model outperformed RGB baselines on registered RGB frames, achieving 96.62% mean matching accuracy and 67.18% mean average accuracy at 10 degree on pose estimation, demonstrating consistent improvements across multiple evaluation metrics. Conclusion: Integrating spectral information from an HSI cube offers a promising approach for robust monocular 3D reconstruction in MIS, addressing limitations of texture-poor surgical environments through enhanced spectral-spatial feature discrimination. Our model and dataset are available at https://github.com/alexsaikia/HyKey-Hyperspectral-Keypoint-Detection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。