arXiv:2509.13175cs.CV2025-09被引 7

用大模型自动标注医学影像,低成本实现高性能视觉语言对齐。

More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era

  • 用大模型自动提取放射科报告中的诊断标签,准确率超96%。
  • 仅用3D ResNet-18和普通CLIP训练,零样本诊断AUC达83.8%。
  • 适合想低成本构建医疗多模态模型的研究者与开发者。

大语言模型的兴起为医学对比视觉语言预训练带来全新机遇。本文表明,现代大语言模型可无需复杂提示工程,精准自动提取放射科报告中的诊断标签(实验中AUC > 96%),从而以极低成本(约3美元/5万对CT图像-报告)构建大规模“准标准”数据集。基于该数据集训练的视觉编码器性能媲美使用专用BERT模型提取标签的结果,推动了大规模有监督预训练的普及。进一步发现,有监督预训练显著提升视觉语言对齐能力。本方法仅使用3D ResNet-18与原始CLIP训练,即在CT-RATE上实现83.8% AUC的零样本诊断,在RAD-ChestCT上达77.3% AUC,跨模态检索指标也显著提升(image-image MAP@50=53.7%,report-image Recall@100=52.2%)。结果证明,利用大模型可构建更高效、可扩展的医疗AI系统。代码已开源。

原文摘要 · Abstract (English)

The emergence of Large Language Models (LLMs) presents unprecedented opportunities to revolutionize medical contrastive vision-language pre-training. In this paper, we show how LLMs can facilitate large-scale supervised pre-training, thereby advancing vision-language alignment. We begin by demonstrate that modern LLMs can automatically extract diagnostic labels from radiology reports with remarkable precision (>96\% AUC in our experiments) without complex prompt engineering, enabling the creation of large-scale "silver-standard" datasets at a minimal cost (~\$3 for 50k CT image-report pairs). Further, we find that vision encoder trained on this "silver-standard" dataset achieves performance comparable to those trained on labels extracted by specialized BERT-based models, thereby democratizing the access to large-scale supervised pre-training. Building on this foundation, we proceed to reveal that supervised pre-training fundamentally improves contrastive vision-language alignment. Our approach achieves state-of-the-art performance using only a 3D ResNet-18 with vanilla CLIP training, including 83.8\% AUC for zero-shot diagnosis on CT-RATE, 77.3\% AUC on RAD-ChestCT, and substantial improvements in cross-modal retrieval (MAP@50=53.7\% for image-image, Recall@100=52.2\% for report-image). These results demonstrate the potential of utilizing LLMs to facilitate {\bf more performant and scalable} medical AI systems. Our code is avaiable at https://github.com/SadVoxel/More-performant-and-scalable.

医学影像多模态大模型应用视觉语言对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。