让3D医学影像与报告对齐更精准,按解剖结构分块训练。
Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning

- 按器官分块提取图像-文本匹配查询,实现局部对齐
- 在多个任务上超越CLIP基线,最高提升12.7%准确率
- 无需标注即可自动生成注意力热图,适合医疗可解释性研究
视觉-语言对比预训练已成为3D医学基础模型的主流方法,利用临床中大量成对的影像与报告数据。然而,医学图像通常涵盖数十个器官,放射科报告远长于自然图像描述,且包含多个结构化段落。传统CLIP式预训练将每种模态压缩为单一全局向量,易丢失关键细节。我们提出ConQuer(概念查询),在CLIP全局对齐基础上增加一组局部对齐,每个概念对应一个查询。ConQuer将报告拆分为特定概念的段落,学习跨注意力查询以聚合匹配的图像特征,无需分割掩码或空间监督。每个概念独立进行对比学习。概念可为任意语义定位单元,此处为解剖区域,每器官一个查询。作为副产物,每个查询学习到聚焦于其概念的注意力图,提供内置的空间可解释性。我们用ConQuer训练了Jolia,一个在胸部和腹部CT上预训练的3D CT基础模型。Jolia在病灶分类、报告生成及跨中心迁移任务上持续优于CLIP基线,并在多个公开基准上达到新SOTA。Jolia权重已发布于https://huggingface.co/raidium/Jolia。
原文摘要 · Abstract (English)
Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP's global alignment with a set of localized alignments, one per concept. ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia's weights are available at https://huggingface.co/raidium/Jolia
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。