用语言模型提升医学图像分割的效率与泛化能力
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
- 通过概率跨模态注意力融合图文特征，实现细粒度文本引导分割
- 在16个数据集上显著提升精度与鲁棒性，且仅需少量标注数据
- 输出可解释的不确定性图，适合临床辅助决策场景
医学图像分割因标注数据有限、解剖结构模糊及域偏移问题而面临挑战。尽管视觉-语言模型如CLIP具备强大的跨模态表征能力，但其在密集、文本驱动的医学图像分割中的潜力仍待挖掘。本文提出MedCLIPSeg，一种针对医学图像分割的鲁棒、高效且具备不确定性建模能力的新框架。该方法利用片段级CLIP嵌入，通过概率跨模态注意力实现图像与文本标记间的双向交互，并显式建模预测不确定性。结合软片段级对比损失，促进多样文本提示下的细粒度语义学习，显著提升数据效率与域泛化能力。在涵盖五种成像模态和六种器官的16个数据集上进行的广泛实验表明，MedCLIPSeg在准确率、效率和鲁棒性方面均优于现有方法，同时生成可解释的不确定性图，揭示分割结果的局部可靠性。本工作展示了概率视觉-语言建模在文本驱动医学图像分割中的潜力。
原文摘要 · Abstract (English)
Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。