通过动态选提示提升医学多模态模型跨域泛化能力
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
- 用可学习提示库+低熵准则动态选择最佳提示
- 少样本下在11个数据集上达到新SOTA
- 适合医学图像与文本跨模态任务研究者
预训练的生物医学视觉语言模型（如BioMedCLIP）在平均性能上表现良好，但在类别间差异小、采集方式差异显著的挑战性模态上性能下降，尤其在少样本设置下且模态先验与预训练语料差异较大时。我们提出BioVLM，一种无需大量主干网络微调的提示学习框架。BioVLM学习一个多样化的提示库，并引入动态提示选择机制：对每个输入，基于预测分布的低熵准则选择最具判别性的提示，有效结合稀疏少样本证据与大型语言模型的丰富语义先验。为增强这种耦合，我们蒸馏高置信度的LLM属性，并通过强/弱增强一致性强制鲁棒知识迁移。测试时，BioVLM通过选择适配模态的提示实现自适应，支持未见类别和领域的迁移，同时保持训练轻量、推理高效。在11个MedMNIST+2D数据集上，BioVLM在三种不同泛化设置下均达到新SOTA。代码已开源。
原文摘要 · Abstract (English)
Pretrained biomedical vision-language models (VLMs) such as BioMedCLIP perform well on average but often degrade on challenging modalities where inter-class margins are small and acquisition-specific variations are pronounced, especially under few-shot supervision and when modality priors differ from pretraining corpora substantially. We propose BioVLM, a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning. BioVLM learns a diverse prompt bank and introduces dynamic prompt selection: for each input, it selects the most discriminative prompts via a low-entropy criterion on the predictive distribution, effectively coupling sparse few-shot evidence with rich LLM semantic priors. To strengthen this coupling, we distill high-confidence LLM-derived attributes and enforce robust knowledge transfer through strong/weak augmentation consistency. At test time, BioVLM adapts by choosing modality-appropriate prompts, enabling transfer to unseen categories and domains, while keeping training lightweight and inference efficient. On 11 MedMNIST+ 2D datasets, BioVLM achieves new state of the art across three distinct generalization settings. Codes are available at https://github.com/mainaksingha01/BioVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。