实时生成外科手术物理模拟,支持人机与智能体即时交互。
NVIDIA Cosmos-H-Dreams: Real-Time Generative Physics Simulation for Surgical Robotics
- 用动作条件生成模型+师生蒸馏,实现高帧率可控仿真
- 单卡实现在160帧/秒下运行,支持多种控制接口
- 首个支持实时人类与策略控制的交互式手术世界模型
外科手术生成式仿真仍缺乏实时交互能力。传统物理-机器人实验耗时长、成本高且难以复现,而经典仿真器难以同时实现逼真视觉效果与可变形组织动态。我们提出Cosmos-H-Dreams,一个集成的实时外科世界模型系统,结合动作条件生成模型、师生蒸馏方法及基于NVIDIA FlashDreams流推理库的部署栈。在Cosmos-H-Surgical-Simulator基础上,使用特定实体与手术数据进行后训练,通过自强迫蒸馏将双向教师模型压缩为因果性、少步数的学生模型,使被动视频生成器变为可控制的实时仿真器,在单张NVIDIA RTX PRO 6000 Blackwell工作站GPU上达到约160帧/秒。关键优势在于控制器无关:任意输出机器人运动学序列的接口均可驱动它。我们演示了通过浏览器键盘经WebRTC、Meta Quest头显经WebXR、CMR Surgical Versius等商用手术机器人控制台,以及闭环运行的学习策略进行实时控制。据我们所知,这是首个支持实时人类与策略控制的交互式外科世界模型,为手术教育、合成数据生成和术中决策支持提供开放平台。
原文摘要 · Abstract (English)
Generative simulation for surgical robotics still lacks real-time interaction. Physical-robot experiments, often involving animal or cadaver labs, are time-consuming, costly, and difficult to reproduce, while classical simulators struggle to capture photorealistic appearance and deformable-tissue dynamics. We address this gap with Cosmos-H-Dreams, an integrated real-time surgical world-model system combining an action-conditioned generative model, a teacher-to-student distillation recipe, and a deployment stack built on the NVIDIA FlashDreams streaming-inference library. Starting from Cosmos-H-Surgical-Simulator, a multi-embodiment action-conditioned surgical video world model fine-tuned on the large-scale Open-H-Embodiment corpus, we post-train this checkpoint on embodiment- and procedure-specific data. By distilling the resulting bidirectional teacher into a causal, few-step student with Self Forcing, we turn a passive video generator into a controllable surgical simulator that streams at $\sim$160 inference FPS on a single NVIDIA RTX PRO 6000 Blackwell workstation GPU. Crucially, Cosmos-H-Dreams is controller-agnostic: any interface that emits a stream of robot kinematics can drive it. We demonstrate live control through a browser keyboard over WebRTC, a Meta Quest headset over WebXR, a commercial surgical robot console such as CMR Surgical's Versius, and learned policies operating in closed loop. To our knowledge, this is the first interactive surgical world model supporting live human and policy control. Human operators and policies alike can act inside the synthesized world and observe the consequences in real time. We release Cosmos-H-Dreams as an open surgical simulation system, providing a common foundation for surgical education, scalable synthetic data generation, and future intraoperative decision support.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。