用世界模型实现手术机器人策略的自动在线评估,省时省力且结果可靠。
Cosmos-Surg-dVRK: World Foundation Model-based Automated Online Evaluation of Surgical Robot Policy Learning
- 基于宇宙世界模型微调手术专用模型,支持高保真仿真
- 在真实dVRK平台上验证,自动化评估与人工评价高度一致
- 适合需要快速迭代手术机器人策略的研究者使用
手术机器人与视觉-语言-动作模型的发展推动了自主手术策略的进步和高效评估方法的出现。然而,在da Vinci Research Kit(dVRK)等物理平台直接评估这些策略仍受限于高昂成本、时间消耗、可复现性差及执行变异。物理人工智能的世界基础模型(WFM)为高保真模拟复杂现实手术任务(如软组织变形)提供了变革性路径。本文提出Cosmos-Surg-dVRK,即对Cosmos WFM进行手术领域微调,并结合训练好的视频分类器,实现手术策略的全自动在线评估与基准测试。我们在两个不同的手术数据集上进行了评估:在桌面缝合垫任务中,Cosmos-Surg-dVRK中的在线回放与真实dVRK Si平台上的策略表现高度相关,且人类标注者与基于V-JEPA 2的视频分类器判断结果一致。初步的离体猪胆囊切除实验也显示出与真实评估的良好对齐,表明该平台在更复杂手术中的潜力。
原文摘要 · Abstract (English)
The rise of surgical robots and vision-language-action models has accelerated the development of autonomous surgical policies and efficient assessment strategies. However, evaluating these policies directly on physical robotic platforms such as the da Vinci Research Kit (dVRK) remains hindered by high costs, time demands, reproducibility challenges, and variability in execution. World foundation models (WFM) for physical AI offer a transformative approach to simulate complex real-world surgical tasks, such as soft tissue deformation, with high fidelity. This work introduces Cosmos-Surg-dVRK, a surgical finetune of the Cosmos WFM, which, together with a trained video classifier, enables fully automated online evaluation and benchmarking of surgical policies. We evaluate Cosmos-Surg-dVRK using two distinct surgical datasets. On tabletop suture pad tasks, the automated pipeline achieves strong correlation between online rollouts in Cosmos-Surg-dVRK and policy outcomes on the real dVRK Si platform, as well as good agreement between human labelers and the V-JEPA 2-derived video classifier. Additionally, preliminary experiments with ex-vivo porcine cholecystectomy tasks in Cosmos-Surg-dVRK demonstrate promising alignment with real-world evaluations, highlighting the platform's potential for more complex surgical procedures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。