双模态提示调优提升医学图像分类，兼顾文本与图像结构信息。
Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models
- 结合临床模板与大模型生成的双文本提示，提取领域知识。
- 引入零向量软提示重分配注意力，避免关注非诊断区域。
- 在11个数据集上平均准确率66.14%，显著优于现有方法。
提示学习是适应预训练视觉语言模型（VLMs）进行少样本医学图像分类的有效范式。然而，现有方法多仅使用文本提示，忽略了医学图像中复杂的解剖结构和细微病理特征。本文提出Biomed-DPT，一种增强知识的双模态提示调优方法。在文本提示设计中，构建包含模板驱动的临床提示与大语言模型（LLM）驱动的领域自适应提示的双重提示，并通过知识蒸馏技术从领域自适应提示中提取临床知识。在视觉提示设计中，引入零向量作为软提示，实现注意力重分配，从而避免对非诊断区域的关注及对非关键病理特征的误识别。Biomed-DPT在覆盖9种模态和10个器官的11个生物医学图像数据集上实现了平均分类准确率66.14%，基础类达到78.06%，新类达到75.97%，分别优于上下文优化（CoOp）方法6.20%、3.78%和8.04%。代码已开源。
原文摘要 · Abstract (English)
Prompt learning is one of the most effective paradigms for adapting pre-trained vision-language models (VLMs) to the biomedical image classification tasks in few shot scenarios. However, most of the current prompt learning methods only used the text prompts and ignored the particular structures (such as the complex anatomical structures and subtle pathological features) in the biomedical images. In this work, we propose Biomed-DPT, a knowledge-enhanced dual modality prompt tuning technique. In designing the text prompt, Biomed-DPT constructs a dual prompt including the template-driven clinical prompts and the large language model (LLM)-driven domain-adapted prompts, then extracts the clinical knowledge from the domain-adapted prompts through the knowledge distillation technique. In designing the vision prompt, Biomed-DPT introduces the zero vector as a soft prompt to leverage attention re-weighting so that the focus on non-diagnostic regions and the recognition of non-critical pathological features are avoided. Biomed-DPT achieves an average classification accuracy of 66.14\% across 11 biomedical image datasets covering 9 modalities and 10 organs, with performance reaching 78.06\% in base classes and 75.97\% in novel classes, surpassing the Context Optimization (CoOp) method by 6.20\%, 3.78\%, and 8.04\%, respectively. Our code are available at \underline{https://github.com/Kanyooo/Biomed-DPT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。