医学影像领域用文本训练视觉语言模型效果有限，单模态预训练反而更靠谱。
A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
- 改用细粒度标签的单模态预训练方法
- 在多源异构数据上表现优于当前主流视觉语言模型
- 适合需要精确标注的医学影像分析任务
视觉-语言预训练近年来因可利用大规模数据学习丰富特征表示而受到关注，并迅速进入医学图像分析领域。尽管已有大量研究致力于构建放射科视觉-语言模型，但具备图像-文本标注的医学数据集稀缺，且医学概念精细，现有模型难以有效编码。本文提出回归基础，采用细粒度标签进行监督的单模态预训练，通过广泛对比发现其性能极具竞争力，更适合整合异构数据源。结果也质疑了近期视觉-语言模型在开放词汇泛化上的潜力，因其评估设置过于乐观。最后，我们探索了融合细粒度标签与噪声文本监督的新方法。
原文摘要 · Abstract (English)
Vision-language pre-training has recently gained popularity as it allows learning rich feature representations using large-scale data sources. This paradigm has quickly made its way into the medical image analysis community. In particular, there is an impressive amount of recent literature developing vision-language models for radiology. However, the available medical datasets with image-text supervision are scarce, and medical concepts are fine-grained, involving expert knowledge that existing vision-language models struggle to encode. In this paper, we propose to take a prudent step back from the literature and revisit supervised, unimodal pre-training, using fine-grained labels instead. We conduct an extensive comparison demonstrating that unimodal pre-training is highly competitive and better suited to integrating heterogeneous data sources. Our results also question the potential of recent vision-language models for open-vocabulary generalization, which have been evaluated using optimistic experimental settings. Finally, we study novel alternatives to better integrate fine-grained labels and noisy text supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。