用大模型提升医学图文对齐,解决术语复杂与幻觉问题
SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation

- 将生成式大模型改造为语义均衡的编码器,适配临床报告
- 通过离线缓存实现高效训练,减少内存开销
- 新增解剖位置与否定词感知损失,降低医学幻觉
视觉-语言预训练(VLP)是医学多模态表征学习的核心。现有框架受限于轻量级文本编码器的上下文窗口小和表征能力弱,难以处理长篇、术语密集的临床报告。虽然引入医学大语言模型(LLM)可增强临床推理能力,但带来三大瓶颈:(i) 生成式LLM在标准对比学习下出现表征异构性坍塌;(ii) 大批次联合端到端训练导致内存开销巨大;(iii) 原始对比损失忽略细微解剖侧位与否定修饰,引发医学幻觉。为此,我们提出SCALPEL——一种基于大模型驱动编码器学习的语义跨模态对齐框架。首先,通过临床报告对比微调,将生成式LLM转化为同质化编码器;其次,采用非对称对齐策略结合离线特征缓存,实现高效训练;关键创新在于设计解剖-否定感知目标,显式惩罚涉及侧位混淆或错误否定的图文不匹配。在MIMIC-CXR、CheXpert和IU X-Ray多个基准上实验表明,SCALPEL在跨模态检索、零样本疾病分类与医学视觉问答任务中均达到当前最优性能。
原文摘要 · Abstract (English)
Vision-language pre-training (VLP) serves as a cornerstone for medical multimodal representation learning. However, existing medical VLP frameworks are often constrained by the limited context windows and shallow representational capacities of lightweight text encoders when processing lengthy, terminology-dense clinical reports. While integrating medical large language models (LLMs) offers unprecedented clinical reasoning capabilities, it introduces three major bottlenecks: (i) the anisotropic representational collapse of generative LLMs under standard contrastive objectives, (ii) the prohibitive memory overhead of joint end-to-end training with large batch sizes, and (iii) the medical hallucinations induced by vanilla contrastive losses that ignore fine-grained anatomical laterality and negation modifiers. To address these challenges, we propose \textbf{SCALPEL}, a \textbf{S}emantic \textbf{C}ross-modal \textbf{A}lignment framework via \textbf{L}LM-\textbf{P}owered \textbf{E}ncoder \textbf{L}earning. First, Clinical Report Contrastive fine-tuning converts a generative LLM into an isotropic encoder via domain-specific clinical text adaptation. Second, an asymmetric alignment strategy leverages offline feature caching to enable efficient training. Critically, we formulate an Anatomy-Negation Aware Objective that explicitly penalizes mismatched image-text pairs involving laterality confusion or false negations. Extensive experiments across MIMIC-CXR, CheXpert, and IU X-Ray benchmarks demonstrate that SCALPEL achieves state-of-the-art performance in cross-modal retrieval, zero-shot disease classification and medical visual question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。