通过文本自监督增强医学影像报告生成的语义对齐。
TextSLIP: Text Self-Supervised CLIP for Medical Report Generation

- 在CLIP基础上引入文本内部对比学习,提升文本表征区分度。
- 在700万脑部MRI图文对上预训练,报告生成指标持续优于基线。
- 适合需要细粒度语言监督的医学多模态任务研究者。
自动化放射科报告生成对提升报告一致性与临床流程效率至关重要。尽管对比语言-图像预训练(CLIP)推动了医学视觉语言建模的发展,但现有CLIP类方法在复杂报告生成中仍缺乏细粒度语义监督。标准CLIP主要优化跨模态对齐,未显式构建指导视觉表示学习的文本嵌入空间。为此,我们提出TextSLIP,一种通用医学视觉语言预训练框架,通过引入模内文本对比学习增强CLIP。通过自监督生成的文本增强对提升文本嵌入可区分性,使视觉编码器获得更精细的语言监督。初步验证中,我们在700万张脑部MRI图像-文本对数据集上预训练TextSLIP,并在报告生成架构中微调预训练视觉编码器。与基于CLIP的基线相比,TextSLIP在报告生成指标上表现一致更优。消融实验表明,文本侧自监督学习对性能提升有贡献。结果表明,文本级对比学习是改善医学视觉-文本对齐的有前景方向,未来需在更多医学领域进行验证。
原文摘要 · Abstract (English)
Automating radiology report generation is important for improving reporting consistency and clinical workflows . While Contrastive Language--Image Pretraining (CLIP) has advanced medical vision language modeling, existing CLIP-style approaches may still provide insufficient fine-grained semantic supervision for complex report generation. Standard CLIP primarily optimizes cross-modal alignment, without explicitly structuring the textual embedding space that guides visual representation learning. To address this limitation, we propose TextSLIP, a general medical vision-language pretraining framework that augments CLIP with intra-modal text contrastive learning. By improving textual embedding discriminability through self-supervised augmented text pairs, TextSLIP is designed to provide finer-grained linguistic supervision to the visual encoder. As an initial validation, we pretrain TextSLIP on a curated dataset of 7 million brain MRI image-text pairs and fine-tune the pretrained visual encoder within a report generation architecture. In controlled comparisons with CLIP-style baselines, TextSLIP shows consistent improvements on report generation metrics. Ablation studies further suggest that text-side self-supervision contributes to the observed gains. These results indicate that text-level contrastive learning is a promising direction for improving medical visual-textual alignment, while broader validation across additional medical domains remains an important next step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。