提出首个医学视觉语言模型合并基准,用胜者为王法提升多专家模型融合效果。
Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach

- 采用胜者为王策略,保留专家模型中主导参数,舍弃弱参数。
- 在8种影像模态、16个微调模型上验证,性能超越现有方法。
- 适合需部署多个医疗模型但资源受限的研究与临床场景。
大型视觉语言模型(LVLM)可通过低秩适配(LoRA)等参数高效微调方法适应特定医学影像任务,形成针对不同成像模态和临床场景的专家模型体系。然而,实际部署多个专家模型会带来显著计算与运维开销。模型合并可在不重新训练的情况下将多个专家模型融合为单一模型,是潜在解决方案,但在医学领域仍鲜有研究。本文首次系统研究医学LVLM的模型合并问题,提出MergeMedBench基准,涵盖8种影像模态和多样临床任务类型,包含基于两个主流架构的16个LoRA微调模型。我们全面评估现有合并方法,并提出一种简单、无需超参数的胜者为王策略:仅保留各专家模型中最具主导性的参数,舍弃较弱参数。该方法避免了平均或对齐类策略带来的信息稀释问题。尽管结构简单,胜者为王在各项指标上持续优于现有方法,为LoRA合并提供了新视角,并为未来研究提供强效实用基线。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) can be adapted to specialized medical imaging tasks via parameter-efficient fine-tuning approaches such as low-rank adaptation (LoRA), leading to a growing ecosystem of expert models tailored to specific imaging modalities and clinical scenarios. However, deploying multiple expert LVLMs in practice incurs substantial computational and operational overhead. Model merging provides a promising solution by consolidating multiple experts into a single model without retraining, yet it remains largely unexplored in the medical domain. In this work, we present the first systematic study of model merging for medical LVLMs. We introduce MergeMedBench, a comprehensive benchmark spanning eight imaging modalities and diverse clinical task types, comprising 16 LoRA fine-tuned models built upon two mainstream architectures. We conduct an extensive evaluation of existing merging methods and further propose winner-take-all, a simple and hyperparameter-free approach that retains only the most dominant parameters across expert models. By preserving the critical parameters that govern model behavior and discarding weaker ones, our method avoids the information dilution inherent in averaging- or alignment-based strategies. Despite its simplicity, winner-take-all consistently outperforms existing approaches, offering both a new perspective on LoRA merging and a strong practical baseline for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。