arXiv:2509.06831cs.CV2025-09中稿 · ML-CDS @ MICCAI 20…

用通用模型提升手术数据分析,融合视频与术中多源数据效果更佳。

Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis

  • 用V-JEPA模型在手术视频上微调,实现领域自适应。
  • 在自建数据集上,多模态融合使术后住院时长预测准确率提升。
  • 开源代码与模型,适合医疗AI研究者使用。

本文研究如何通过迁移学习适配通用基础模型,并融合手术室内的多模态数据,以支持微创手术的数据分析。采用V-JEPA作为单模态基础模型,在自建肝切除手术视频数据集上,分析预测住院时长和术后并发症的任务;在公开的HeiCo数据集上,分析手术阶段识别任务。以预训练的V-JEPA为基线,先在未标注的保留视频上进行微调,评估领域适应后的性能变化;再基于模块化决策支持网络思想,引入手术室中的时间序列数据流,通过独立编码器构建与V-JEPA嵌入共享的表示空间。实验表明,域内数据微调可提升模型表现;在自建数据集上,融合额外时序数据进一步改善结果;在HeiCo数据集上,纯视频基线模型精度达到EndoVis2017挑战赛最优提交水平,微调后精度进一步提高。结果证明,手术数据科学可有效利用公开通用基础模型,且领域适配与多模态融合具有显著潜力。代码与模型权重已开源。

原文摘要 · Abstract (English)

We investigate how both the adaptation of a generic foundation model via transfer learning and the integration of complementary modalities from the operating room (OR) can support surgical data science. To this end, we use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support. We analyze how the model's downstream performance can benefit (a) from finetuning on unlabeled surgical video data and (b) from providing additional time-resolved data streams from the OR in a multimodal setup. In an in-house dataset of liver surgery videos, we analyze the tasks of predicting hospital length of stay and postoperative complications. In videos of the public HeiCo dataset, we analyze the task of surgical phase recognition. As a baseline, we apply pretrained V-JEPA to all tasks. We then finetune it on unlabeled, held-out videos to investigate its change in performance after domain adaptation. Following the idea of modular decision support networks, we integrate additional data streams from the OR by training a separate encoder to form a shared representation space with V-JEPA's embeddings. Our experiments show that finetuning on domain-specific data increases model performance. On the in-house data, integrating additional time-resolved data likewise benefits the model. On the HeiCo data, accuracy of the pretrained video-only, single-modality baseline setup is on par with the top-performing submissions of the EndoVis2017 challenge, while finetuning on domain-specific data increases accuracy further. Our results thus demonstrate how surgical data science can leverage public, generic foundation models. Likewise, they indicate the potential of domain adaptation and of integrating suitable complementary data streams from the OR. To support further research, we release our code and model weights at https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025.

手术分析多模态基础模型迁移学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。