用因果干预提升CLIP在医学图像分割中的表现
CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Intervention
- 通过跨模态解码将CLIP语义空间适配医学图像
- 自标注混杂因素并挖掘因果特征，提升分割准确率
- 适合医学图像分析与视觉语言模型应用的研究者
针对文本描述引导的医学图像分割任务，本文提出CausalCLIPSeg框架，利用未在医学数据上训练的CLIP模型实现文本到像素的对齐。通过定制化的跨模态解码方法，将CLIP丰富的语义空间迁移到医学领域。为缓解混杂偏差导致的虚假相关性学习，引入因果干预模块，自动标注混杂因素，并从输入中提取因果特征用于分割判断。同时设计对抗式极小极大博弈，优化因果特征并惩罚混杂特征。大量实验验证了该方法的领先性能。代码已公开于https://github.com/WUTCM-Lab/CausalCLIPSeg。
原文摘要 · Abstract (English)
Referring medical image segmentation targets delineating lesions indicated by textual descriptions. Aligning visual and textual cues is challenging due to their distinct data properties. Inspired by large-scale pre-trained vision-language models, we propose CausalCLIPSeg, an end-to-end framework for referring medical image segmentation that leverages CLIP. Despite not being trained on medical data, we enforce CLIP's rich semantic space onto the medical domain by a tailored cross-modal decoding method to achieve text-to-pixel alignment. Furthermore, to mitigate confounding bias that may cause the model to learn spurious correlations instead of meaningful causal relationships, CausalCLIPSeg introduces a causal intervention module which self-annotates confounders and excavates causal features from inputs for segmentation judgments. We also devise an adversarial min-max game to optimize causal features while penalizing confounding ones. Extensive experiments demonstrate the state-of-the-art performance of our proposed method. Code is available at https://github.com/WUTCM-Lab/CausalCLIPSeg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。