arXiv:2604.18134cs.CV2026-04中稿 · CVPR

用大模型自动生成手术视频文本,解决标注成本难题

Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?

论文配图:Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?
图 1 · 摘自论文原文
  • 用大模型自动生成手术视频字幕,实现无专家标注的多模态训练
  • 在AutoLaparo和Cholec80上实现可比零样本对齐性能
  • 通过动态降权不确定文本,保护视觉模型医学先验

自监督学习已催生强大的手术视觉编码器,具备时空理解能力。但将其扩展至多模态推理任务受限于专家文本标注的高昂成本。为突破这一瓶颈,我们提出LIME——一个基于公开手术视频、无需人工标注的大规模多模态数据集,其文本由大语言模型(LLM)生成。尽管LIME具备极强可扩展性,但生成文本可能包含幻觉等错误,可能导致标准对比学习中预训练医学先验严重退化。为此,我们提出SurgLIME:一种参数高效的视觉-语言预训练框架,利用噪声文本进行可靠跨模态对齐。SurgLIME采用LoRA适配的双编码器架构,保留基础医学先验,并引入自动化置信度估计机制,在对比对齐过程中动态降低不确定文本权重。在AutoLaparo和Cholec80基准上的评估显示,SurgLIME实现了具有竞争力的零样本跨模态对齐,同时保持了视觉基础模型的稳健线性探测性能。数据集、代码与模型已开源:https://github.com/visurg-ai/SurgLIME。

原文摘要 · Abstract (English)

Recent advancements in self-supervised learning have led to powerful surgical vision encoders capable of spatiotemporal understanding. However, extending these visual foundations to multi-modal reasoning tasks is severely bottlenecked by the prohibitive cost of expert textual annotations. To overcome this scalability limitation, we introduce \textbf{LIME}, a large-scale multi-modal dataset derived from open-access surgical videos using human-free, Large Language Model (LLM)-generated narratives. While LIME offers immense scalability, unverified generated texts may contain errors, including hallucinations, that could potentially lead to catastrophically degraded pre-trained medical priors in standard contrastive pipelines. To mitigate this, we propose \textbf{SurgLIME}, a parameter-efficient Vision-Language Pre-training (VLP) framework designed to learn reliable cross-modal alignments using noisy narratives. SurgLIME preserves foundational medical priors using a LoRA-adapted dual-encoder architecture and introduces an automated confidence estimation mechanism that dynamically down-weights uncertain text during contrastive alignment. Evaluations on the AutoLaparo and Cholec80 benchmarks show that SurgLIME achieves competitive zero-shot cross-modal alignment while preserving the robust linear probing performance of the visual foundation model. Dataset, code, and models are publicly available at https://github.com/visurg-ai/SurgLIME.

多模态手术视觉大模型生成自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。