arXiv:2409.01003cs.CV2024-09被引 1

无需固定相机位姿,实现动态手术视频的高保真重建

Free-DyGS: Camera-Pose-Free Scene Reconstruction for Dynamic Surgical Videos with Gaussian Splatting

  • 用高斯点云结合逐帧优化,自适应处理移动镜头与组织变形
  • 在两个真实手术数据集上,重建精度和速度均优于现有方法
  • 适合需要真实场景重建的外科导航与教学应用

高保真手术场景重建对术中导航、外科教学等应用至关重要。然而,现有方法通常假设理想场景:要么仅处理形变组织但要求固定相机位姿,要么允许镜头移动但只重建静态场景。本文针对更真实却更具挑战性的设定——移动相机下的高度动态手术场景,首次引入高斯点云(Gaussian Splatting, GS)技术,提出新框架 Free-DyGS,实现自由位姿重建。该模型采用预训练的稀疏高斯回归器(SGR)进行高效场景初始化;每帧通过联合优化形变模型与6D相机位姿,利用相邻帧间形变差异小的特点加速训练;引入场景扩展机制以覆盖移动相机带来的新区域;并设计回溯形变重演(RDR)策略,在逐帧训练中保持整段视频的形变一致性。在 StereoMIS 与 Hamlyn 两个数据集上的实验表明,Free-DyGS 在渲染精度与效率上均超越现有先进方法。

原文摘要 · Abstract (English)

High-fidelity reconstruction of surgical scene is a fundamentally crucial task to support many applications, such as intra-operative navigation and surgical education. However, most existing methods assume the ideal surgical scenarios - either focus on dynamic reconstruction with deforming tissue yet assuming a given fixed camera pose, or allow endoscope movement yet reconstructing the static scenes. In this paper, we target at a more realistic yet challenging setup - free-pose reconstruction with a moving camera for highly dynamic surgical scenes. Meanwhile, we take the first step to introduce Gaussian Splitting (GS) technique to tackle this challenging setting and propose a novel GS-based framework for fast reconstruction, termed \textit{Free-DyGS}. Concretely, our model embraces a novel scene initialization in which a pre-trained Sparse Gaussian Regressor (SGR) can efficiently parameterize the initial attributes. For each subsequent frame, we propose to jointly optimize the deformation model and 6D camera poses in a frame-by-frame manner, easing training given the limited deformation differences between consecutive frames. A Scene Expansion scheme is followed to expand the GS model for the unseen regions introduced by the moving camera. Moreover, the framework is equipped with a novel Retrospective Deformation Recapitulation (RDR) strategy to preserve the entire-clip deformations throughout the frame-by-frame training scheme. The efficacy of the proposed Free-DyGS is substantiated through extensive experiments on two datasets: StereoMIS and Hamlyn datasets. The experimental outcomes underscore that Free-DyGS surpasses other advanced methods in both rendering accuracy and efficiency. Code will be available.

三维重建手术视频高斯点云动态场景

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。