双提示增强医学图像描述，提升模态识别与临床相关性
DualPrompt-MedCap: A Dual-Prompt Enhanced Approach for Medical Image Captioning
- 用两个专用提示：模态感知提示+问题引导提示，增强大模型理解
- 模态识别准确率比基线高22%，生成描述更完整且对齐临床问题
- 适合需要精准医学报告生成的临床辅助系统研发者
基于视觉-语言模型的医学图像描述在临床诊断辅助中展现出巨大潜力。然而，生成上下文相关且准确识别模态的描述仍具挑战。我们提出DualPrompt-MedCap，一种新型双提示增强框架，通过两个专用组件增强大视觉-语言模型（LVLMs）：(1) 基于半监督分类模型（在医学问答对上预训练）生成的模态感知提示；(2) 利用生物医学语言模型嵌入的问题引导提示。为解决缺乏标注描述的问题，我们还提出一个联合评估框架，同时考虑空间-语义相关性和医学叙述质量。在多个医学数据集上的实验表明，DualPrompt-MedCap相比基线BLIP-3在模态识别准确率上提升22%，并生成更全面、问题对齐的描述。该方法可生成符合临床标准的报告，作为专家先验知识或下游视觉-语言任务的自动标注。
原文摘要 · Abstract (English)
Medical image captioning via vision-language models has shown promising potential for clinical diagnosis assistance. However, generating contextually relevant descriptions with accurate modality recognition remains challenging. We present DualPrompt-MedCap, a novel dual-prompt enhancement framework that augments Large Vision-Language Models (LVLMs) through two specialized components: (1) a modality-aware prompt derived from a semi-supervised classification model pretrained on medical question-answer pairs, and (2) a question-guided prompt leveraging biomedical language model embeddings. To address the lack of captioning ground truth, we also propose an evaluation framework that jointly considers spatial-semantic relevance and medical narrative quality. Experiments on multiple medical datasets demonstrate that DualPrompt-MedCap outperforms the baseline BLIP-3 by achieving a 22% improvement in modality recognition accuracy while generating more comprehensive and question-aligned descriptions. Our method enables the generation of clinically accurate reports that can serve as medical experts' prior knowledge and automatic annotations for downstream vision-language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。