arXiv:2506.08990cs.CVcs.AI2025-06被引 12

用少量参数高效对齐医疗图像与文本,提升检索和零样本分类性能。

Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models

  • 基于掩码建模预训练的视觉模型,通过轻量适配实现跨模态对齐。
  • 在图文检索与零样本分类上分别提升4%和6%准确率,显著优于基线。
  • 适合需要高效医疗多模态理解的研究者与临床辅助系统开发者。

通过跨模态对比学习实现医学图像-文本对齐,在图像-文本匹配任务(如检索与零样本分类)中表现优异。然而,传统基于CLIP的方法在视觉表征能力上存在不足,限制了其对齐效果。相比之下,虽掩码建模预训练模型在直接跨模态匹配上表现不佳,但其视觉表征能力更强。为此,我们提出ALTA(ALign Through Adapting),一种仅需约8%可训练参数、计算开销低于掩码建模预训练1/5的高效医疗视觉-语言对齐方法。ALTA通过适配掩码建模预训练的视觉模型,在图像-文本匹配任务中取得更优性能。此外,引入时间-多视角胸片输入,增强影像与其报告描述间的信息一致性,进一步提升对齐效果。实验表明,ALTA在文本到图像准确率上超越最佳对比模型4%以上,图像到文本检索准确率提升约6%。高效对齐过程还促进了视觉与语言理解能力的提升。代码已公开于https://github.com/DopamineLcy/ALTA。

原文摘要 · Abstract (English)

Medical vision-language alignment through cross-modal contrastive learning shows promising performance in image-text matching tasks, such as retrieval and zero-shot classification. However, conventional cross-modal contrastive learning (CLIP-based) methods suffer from suboptimal visual representation capabilities, which also limits their effectiveness in vision-language alignment. In contrast, although the models pretrained via multimodal masked modeling struggle with direct cross-modal matching, they excel in visual representation. To address this contradiction, we propose ALTA (ALign Through Adapting), an efficient medical vision-language alignment method that utilizes only about 8% of the trainable parameters and less than 1/5 of the computational consumption required for masked record modeling. ALTA achieves superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling. Additionally, we integrate temporal-multiview radiograph inputs to enhance the information consistency between radiographs and their corresponding descriptions in reports, further improving the vision-language alignment. Experimental evaluations show that ALTA outperforms the best-performing counterpart by over 4% absolute points in text-to-image accuracy and approximately 6% absolute points in image-to-text retrieval accuracy. The adaptation of vision-language models during efficient alignment also promotes better vision and language understanding. Code is publicly available at https://github.com/DopamineLcy/ALTA.

医疗多模态视觉语言对齐轻量级模型对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。