让AI像医生一样看胸片,结合病史和注视点提升诊断能力。
Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays
- 融合临床信息与医生注视轨迹,引导模型关注关键区域。
- 在报告生成、零样本分类等任务上超越现有方法,最高提升23.2%。
- 适合医学影像分析、多模态学习研究者参考。
尽管医学视觉-语言预训练取得进展,现有模型仍难以捕捉诊断流程:胸片常被当作无上下文的图像处理,而医生注视轨迹——这一重要视觉推理线索——未被充分建模。这限制了疾病特异性模式的学习,削弱了跨模态对齐。为此,我们提出CoGaze框架,通过上下文增强的视觉编码器,模拟医生如何整合患者病史、症状及诊断意图进行推理;并设计多层级监督机制:(1)通过混合正样本对比学习实现模态内与模态间语义对齐;(2)利用疾病感知的跨模态表示学习注入诊断先验;(3)将医生注视作为概率先验,引导注意力聚焦于诊断相关区域。大量实验表明,CoGaze在多种任务中持续优于当前最优方法,自由文本与结构化报告生成的CheXbertF1最高提升2.0%、BLEU2提升1.2%;零样本分类的AUROC提升23.2%;图像-文本检索的Precision@1提升12.2%。代码已开源。
原文摘要 · Abstract (English)
Despite recent advances in medical vision-language pretraining, existing models still struggle to capture the diagnostic workflow: radiographs are typically treated as context-agnostic images, while radiologists' gaze -- a crucial cue for visual reasoning -- remains largely underexplored by existing methods. These limitations hinder the modeling of disease-specific patterns and weaken cross-modal alignment. To bridge this gap, we introduce CoGaze, a Context- and Gaze-guided vision-language pretraining framework for chest X-rays. We first propose a context-infused vision encoder that models how radiologists integrate clinical context -- including patient history, symptoms, and diagnostic intent -- to guide diagnostic reasoning. We then present a multi-level supervision paradigm that (1) enforces intra- and inter-modal semantic alignment through hybrid-positive contrastive learning, (2) injects diagnostic priors via disease-aware cross-modal representation learning, and (3) leverages radiologists' gaze as probabilistic priors to guide attention toward diagnostically salient regions. Extensive experiments demonstrate that CoGaze consistently outperforms state-of-the-art methods across diverse tasks, achieving up to +2.0% CheXbertF1 and +1.2% BLEU2 for free-text and structured report generation, +23.2% AUROC for zero-shot classification, and +12.2% Precision@1 for image-text retrieval. Code is available at https://github.com/mk-runner/CoGaze.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。