用高斯点云实现可操控的手术器械逼真三维重建
Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splatting
- 基于3D高斯点云与几何预训练，实现器械结构精准建模
- 通过运动学控制与语义高斯追踪，达成逐帧姿态精确估计
- 适用于真实手术视频转仿真，适合医疗AI训练场景
随着手术人工智能与自主化快速发展，真实到仿真（Real2Sim）转换日益重要。本文提出一种新方法 Instrument-Splatting，利用3D高斯点云从单目手术视频中实现完全可控的手术器械三维重建。为兼顾视觉保真度与可操控性，引入几何预训练将高斯点云绑定至部件网格，并定义前向运动学以灵活控制高斯分布。针对无标记视频，设计一种基于语义嵌入高斯的姿势追踪方法，采用渲染-对比机制鲁棒地优化每帧器械姿态与关节状态，使高斯模型能准确学习纹理并实现逼真渲染。在2个公开手术视频及4个离体组织和绿幕采集视频上验证了方法的有效性与优越性。
原文摘要 · Abstract (English)
Real2Sim is becoming increasingly important with the rapid development of surgical artificial intelligence (AI) and autonomy. In this work, we propose a novel Real2Sim methodology, Instrument-Splatting, that leverages 3D Gaussian Splatting to provide fully controllable 3D reconstruction of surgical instruments from monocular surgical videos. To maintain both high visual fidelity and manipulability, we introduce a geometry pre-training to bind Gaussian point clouds on part mesh with accurate geometric priors and define a forward kinematics to control the Gaussians as flexible as real instruments. Afterward, to handle unposed videos, we design a novel instrument pose tracking method leveraging semantics-embedded Gaussians to robustly refine per-frame instrument poses and joint states in a render-and-compare manner, which allows our instrument Gaussian to accurately learn textures and reach photorealistic rendering. We validated our method on 2 publicly released surgical videos and 4 videos collected on ex vivo tissues and green screens. Quantitative and qualitative evaluations demonstrate the effectiveness and superiority of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。