arXiv:2511.12639cs.CV2025-11被引 8

用大模型知识增强提示调优,提升医学图像分类精度。

Medical Knowledge Intervention Prompt Tuning for Medical Image Classification

  • 通过大模型提取病种特征,动态生成个性化提示
  • 在多个数据集上超越现有提示调优方法,最高提升2.8%
  • 适合需要精准疾病识别的医疗影像研究者

视觉-语言基础模型(VLMs)在多种医学下游任务中展现出强大的特征迁移与泛化能力。然而,由于参数量巨大,微调成本高昂。提示调优成为降低内存消耗和训练时间的有效方案,但现有方法难以区分不同医学概念,忽略多模态医学影像中的关键疾病特征。我们发现,经过海量文本训练的大语言模型(LLMs)在提供专业医学知识方面具有优势。为此,提出CILMP方法:将LLMs引入提示调优过程,通过条件干预机制,在低秩线性子空间中提取疾病特异性表示,并生成疾病相关提示。该方法基于单个医学图像进行条件调整,实现实例自适应提示生成。在多个医学图像数据集上的实验表明,CILMP持续优于当前最优提示调优方法,验证了其有效性。代码已公开于https://github.com/usr922/cilmp。

原文摘要 · Abstract (English)

Vision-language foundation models (VLMs) have shown great potential in feature transfer and generalization across a wide spectrum of medical-related downstream tasks. However, fine-tuning these models is resource-intensive due to their large number of parameters. Prompt tuning has emerged as a viable solution to mitigate memory usage and reduce training time while maintaining competitive performance. Nevertheless, the challenge is that existing prompt tuning methods cannot precisely distinguish different kinds of medical concepts, which miss essentially specific disease-related features across various medical imaging modalities in medical image classification tasks. We find that Large Language Models (LLMs), trained on extensive text corpora, are particularly adept at providing this specialized medical knowledge. Motivated by this, we propose incorporating LLMs into the prompt tuning process. Specifically, we introduce the CILMP, Conditional Intervention of Large Language Models for Prompt Tuning, a method that bridges LLMs and VLMs to facilitate the transfer of medical knowledge into VLM prompts. CILMP extracts disease-specific representations from LLMs, intervenes within a low-rank linear subspace, and utilizes them to create disease-specific prompts. Additionally, a conditional mechanism is incorporated to condition the intervention process on each individual medical image, generating instance-adaptive prompts and thus enhancing adaptability. Extensive experiments across diverse medical image datasets demonstrate that CILMP consistently outperforms state-of-the-art prompt tuning methods, demonstrating its effectiveness. Code is available at https://github.com/usr922/cilmp.

医学图像提示调优大模型疾病识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。