不更新模型权重,用精选病例和反思总结提升医疗诊断大模型表现
Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis
- 通过挑选关键病例模拟医生选典型病案
- 在12个医学数据集上达监督模型水平
- 适合无标注资源但需高准确率的医疗场景
通用多模态大模型在医疗诊断中难以捕捉领域特异性特征,性能远低于全监督基线。尽管微调可改善,但专家标注成本高、计算开销大,难以扩展。为在不更新预训练模型参数的前提下弥补差距,我们提出临床医生模拟工作流(Clinician Mimetic Workflow),一种新型上下文学习框架,融合判别性样本核心集选取(DECS)与自精炼经验总结(SRES)。DECS从噪声数据中计算选择具有判别性的视觉核心集,模拟医生参考‘锚定病例’的能力;SRES则通过提炼多样化推理结果,动态构建文本化经验库,模仿临床诊断中的认知与反思过程。在MedMNIST 2D基准全部12个数据集上的实证表明,该方法超越零样本通用及医学大模型,性能接近全监督视觉模型与领域微调的MMLLM,树立了参数高效医疗上下文学习新标杆。
原文摘要 · Abstract (English)
General Multimodal Large Language Models (MLLMs) often underperform in capturing domain-specific nuances in medical diagnosis, trailing behind fully supervised baselines. Although fine-tuning provides a remedy, the high costs of expert annotation and massive computational overhead limit its scalability. To bridge this gap without updating the weights of the pre-trained backbone of the MLLM, we propose a Clinician Mimetic Workflow. This is a novel In-Context Learning (ICL) framework designed to synergize Discriminative Exemplar Coreset Selection (DECS) and Self-Refined Experience Summarization (SRES). Specifically, DECS simulates a clinician's ability to reference "anchor cases" by selecting discriminative visual coresets from noisy data at the computational level; meanwhile, SRES mimics the cognition and reflection in clinical diagnosis by distilling diverse rollouts into a dynamic textual Experience Bank. Extensive evaluation across all 12 datasets of the MedMNIST 2D benchmark demonstrates that our method outperforms zero-shot general and medical MLLMs. Simultaneously, it achieves performance levels comparable to fully supervised vision models and domain-specific fine-tuned MLLMs, setting a new benchmark for parameter-efficient medical in-context learning. Our code is available at an anonymous repository: https://anonymous.4open.science/r/Synergizing-Discriminative-Exemplars-and-Self-Refined-Experience-ED74.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。