arXiv:2605.02126cs.CVcs.LG2026-05

用对比学习对齐超声图像与临床文本,提升零样本迁移能力。

Ultrasound Vision-Language Alignment via Contrastive Learning

论文配图:Ultrasound Vision-Language Alignment via Contrastive Learning
图 1 · 摘自论文原文
  • 构建双编码器框架,将超声图与医学文本映射到统一向量空间。
  • 在16000+图文对上训练,最佳模型跨模态对齐得分达0.682。
  • 模板生成的描述比大模型生成更优,适合临床应用。

超声基础模型在结构化预测任务中表现强劲,但仅依赖视觉信息,难以在缺乏标注的新任务上实现零样本或少样本迁移。为此,我们提出EchoCare-CLIP,一种类CLIP的双编码器对比学习框架,将超声图像与临床文本对齐于共享嵌入空间。我们构建了一个包含乳腺、肝脏、肺和甲状腺的多器官语料库,共超过16,000张图像-文本对,其中78%以上来自专家标注报告,其余通过三阶段模板与大模型生成管道补充。评估涵盖两种文本编码器(CLIP、BioClinicalBERT)和两种标题生成策略(模板、LLM生成),并与OpenAI CLIP及BiomedCLIP对比。结果表明,所训练模型在跨模态对齐上持续优于基线,最优配置达0.682。然而更强对齐不意味着更好下游性能:部分微调的CLIP变体在外部数据集上零样本分类表现最佳(BUSI: 0.709;AULI: 0.626),而全端到端微调因过拟合导致迁移能力下降。线性探针与少样本适应的表现则因数据集而异,反映领域适配与泛化能力间的权衡。进一步发现,模板生成的标题表现不低于甚至优于大模型生成,表明词汇多样性并非质量指标。综上,仅基于公开数据即可实现超声视觉-语言对齐,但稳健临床迁移需平衡领域适配、编码器容量与标注质量。

原文摘要 · Abstract (English)

Ultrasound foundation models have achieved strong performance on structured prediction tasks but remain exclusively vision-based, limiting zero-shot and few-shot transfer to novel tasks where task-specific annotation is scarce. We address this gap with EchoCare-CLIP, a CLIP-style dual-encoder contrastive framework that aligns ultrasound images with clinical text in a shared embedding space. We curate a multi-organ corpus of over 16K image-text pairs spanning breast, liver, lung, and thyroid, with over 78% of captions derived from expert-annotated reports, and complement the remainder with a three-tier template-based and LLM-based caption generation pipeline. We evaluate model configurations spanning two text encoder families (CLIP, BioClinicalBERT) and two caption strategies (template-based, LLM-generated) against OpenAI CLIP and BiomedCLIP baselines. Our trained models consistently improve cross-modal alignment over baselines, with the best configuration achieving a paired alignment score of 0.682. However, stronger alignment does not guarantee better downstream performance: CLIP-based variants with partial fine-tuning achieve the strongest zero-shot classification on external held-out datasets (0.709 on BUSI; 0.626 on AULI), while full end-to-end fine-tuning degrades transfer due to overfitting. On linear probing and few-shot adaptation, model rankings are dataset-dependent, reflecting a trade-off between domain adaptation and representational generalizability. We further show that template-based captions match or outperform LLM-generated captions, suggesting lexical diversity is not a proxy for caption quality. Taken together, our results demonstrate that ultrasound vision-language alignment is achievable from public data alone, but robust clinical transfer requires careful balancing of domain adaptation, encoder capacity, and caption supervision quality.

超声视觉语言对比学习医疗AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。