构建可动态交互的高保真手术室数字孪生,用于智能外科系统训练与测试。
TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research
- 通过实时重建手术室静态结构与人员设备动态行为,融合生成沉浸式3D环境。
- 几何重建精度达厘米级,合成图像与深度数据使模型性能匹配真实世界水平。
- 适合医疗机器人、具身智能等领域的仿真研究,推动临床应用落地。
构建具身智能在智能手术系统中的应用需安全可控的持续学习与评估环境。然而,手术室的安全规范与操作限制阻碍了智能体对真实场景的自由感知与交互。数字孪生提供了高保真、无风险的探索与训练空间。如何构建能捕捉空间、视觉与行为复杂性的动态数字孪生仍是开放挑战。本文提出TwinOR,一种从真实到模拟的基础设施,用于构建逼真的动态手术室数字孪生。该系统重建静态几何结构,并持续建模人员与设备运动。静态与动态成分融合为沉浸式3D环境,支持可控仿真与未来具身探索。框架以厘米级精度重建完整手术室几何结构,同时保留手术流程中的动态交互。实验表明,TwinOR生成立体与单目RGB流及深度观测,用于几何理解与视觉定位任务。使用FoundationStereo和ORB-SLAM3等模型在合成数据上评估,性能处于其在真实室内数据集报告的准确率范围内,证明TwinOR提供的传感器级真实感足以模拟真实世界的感知与定位挑战。通过建立感知基础的真实-仿真流水线,TwinOR实现了手术室动态、高保真的自动数字孪生构建。作为安全可扩展的实验环境,它为将具身智能从仿真向真实临床环境转化开辟了新路径。
原文摘要 · Abstract (English)
Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, safety regulations and operational constraints in operating rooms (ORs) limit agents from freely perceiving and interacting in realistic settings. Digital twins provide high-fidelity, risk-free environments for exploration and training. How we may create dynamic digital representations of ORs that capture relevant spatial, visual, and behavioral complexity remains an open challenge. We introduce TwinOR, a real-to-sim infrastructure for constructing photorealistic and dynamic digital twins of ORs. The system reconstructs static geometry and continuously models human and equipment motion. The static and dynamic components are fused into an immersive 3D environment that supports controllable simulation and facilitates future embodied exploration. The proposed framework reconstructs complete OR geometry with centimeter-level accuracy while preserving dynamic interaction across surgical workflows. In our experiments, TwinOR synthesizes stereo and monocular RGB streams as well as depth observations for geometry understanding and visual localization tasks. Models such as FoundationStereo and ORB-SLAM3 evaluated on TwinOR-synthesized data achieve performance within their reported accuracy ranges on real-world indoor datasets, demonstrating that TwinOR provides sensor-level realism sufficient for emulating real-world perception and localization challenge. By establishing a perception-grounded real-to-sim pipeline, TwinOR enables the automatic construction of dynamic, photorealistic digital twins of ORs. As a safe and scalable environment for experimentation, TwinOR opens new opportunities for translating embodied intelligence from simulation to real-world clinical environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。