无需固定相机位姿,实现动态手术视频的高保真重建
Free-DyGS: Camera-Pose-Free Scene Reconstruction for Dynamic Surgical Videos with Gaussian Splatting
- 用高斯点云结合逐帧优化,自适应处理移动镜头与组织变形
- 在两个真实手术数据集上,重建精度和速度均优于现有方法
- 适合需要真实场景重建的外科导航与教学应用
高保真手术场景重建对术中导航、外科教学等应用至关重要。然而,现有方法通常假设理想场景:要么仅处理形变组织但要求固定相机位姿,要么允许镜头移动但只重建静态场景。本文针对更真实却更具挑战性的设定——移动相机下的高度动态手术场景,首次引入高斯点云(Gaussian Splatting, GS)技术,提出新框架 Free-DyGS,实现自由位姿重建。该模型采用预训练的稀疏高斯回归器(SGR)进行高效场景初始化;每帧通过联合优化形变模型与6D相机位姿,利用相邻帧间形变差异小的特点加速训练;引入场景扩展机制以覆盖移动相机带来的新区域;并设计回溯形变重演(RDR)策略,在逐帧训练中保持整段视频的形变一致性。在 StereoMIS 与 Hamlyn 两个数据集上的实验表明,Free-DyGS 在渲染精度与效率上均超越现有先进方法。
原文摘要 · Abstract (English)
High-fidelity reconstruction of surgical scene is a fundamentally crucial task to support many applications, such as intra-operative navigation and surgical education. However, most existing methods assume the ideal surgical scenarios - either focus on dynamic reconstruction with deforming tissue yet assuming a given fixed camera pose, or allow endoscope movement yet reconstructing the static scenes. In this paper, we target at a more realistic yet challenging setup - free-pose reconstruction with a moving camera for highly dynamic surgical scenes. Meanwhile, we take the first step to introduce Gaussian Splitting (GS) technique to tackle this challenging setting and propose a novel GS-based framework for fast reconstruction, termed \textit{Free-DyGS}. Concretely, our model embraces a novel scene initialization in which a pre-trained Sparse Gaussian Regressor (SGR) can efficiently parameterize the initial attributes. For each subsequent frame, we propose to jointly optimize the deformation model and 6D camera poses in a frame-by-frame manner, easing training given the limited deformation differences between consecutive frames. A Scene Expansion scheme is followed to expand the GS model for the unseen regions introduced by the moving camera. Moreover, the framework is equipped with a novel Retrospective Deformation Recapitulation (RDR) strategy to preserve the entire-clip deformations throughout the frame-by-frame training scheme. The efficacy of the proposed Free-DyGS is substantiated through extensive experiments on two datasets: StereoMIS and Hamlyn datasets. The experimental outcomes underscore that Free-DyGS surpasses other advanced methods in both rendering accuracy and efficiency. Code will be available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。